A World of MQTT on ESP8266

MQTTFrom no-where, I’ve been immersed in the world of MQTT for the past day or two. If you’ve read my previous posts you’ll know I’ve been working on home control for years, mainly as a hobby and so mainly on a budget. I’ve tried several systems out there and eventually settled on doing my own thing because many of the packages have been developed so far then stopped or they’re very complicated or expensive etc..

ESP8266My earlier posts will show you that I’m walking around with a mobile phone with fairly decent control over 3 properties right now, so it’s not that I don’t have solutions – but the controls could be more responsive, I’ve had a hell of a time with the NRF24L01 radios which I plan soon to put up against a wall and shoot and I was in the process of swapping over to some 433Mhz units when out of the blue the ESP8266 modules appeared and changed all that.

Anyway there is one guy developing a network of these things – early days yet and when I re-discovered an MQTT library was available for the ESP8266, it seemed the natural way to do the job (don’t rush – the version out there yesterday has developed a big fault – give it a day or so) – each unit, be it an ESP-01 on it’s own with a sensor – or attached to an Arduino or something more powerful, would be able to send to and received messages from any other unit – marvellous.

The “broker” for messages is the package that accepts messages from devices (or PCs or whatever) and makes them available to other devices. Loads of advice from people to use “free” services – but after wasting many hours – they all come with a catch. Limits on use, lack of guaranteed they’ll actually be up and running or in the case of one – your messages are open for everyone to see. NO THANK YOU.

Petes MQTT TesterThe obvious choice was a local copy of MOSQUITTO – the free package designed, apparently by Roger Light. A simple DOS application which can also act as a service. Marvellous. Now I needed to put the code into the ESP8266 – that worked – marvellous. I even developed my own little client application (well, I tarted up a demo) on Basic 4 Android (B4A). Now I needed some PC client to test the messages… not so marvellous. The command line tools provided with Mosquitto are, well, command line tools – I may be 60 but I’m still part of the pretty-coloured-buttons generation – I really would rather have as little to do with black boxes as possible.  At least half of the links I tried on the web went to projects that haven’t started yet, or old ones that were dead – or just no-where. I finally settled on MQTT-SPY and I have to say.. it’s lovely both in operation and visually. MQTT messages are really trivial comprising a topic – and some content. I set up Mosquitto, subscribed to an arbitrary topic – and sent a message to it. LO AND BEHOLD it appeared at the bottom of MQTT-SPY.

I realised at this point that the messages were utterly non-secure and started to read the documentation for Mosquitto. After a while as I started to ponder cutting my wrists I came to the conclusion that this package was a TAD over the top for my requirements – I got password protection working and tested it out – very nice – but then two of us were using this and in both cases the package failed once.  At that point my friend who was helping me got very excited as he’d discovered RabbitMQ – a VERY pretty alternative with a nice shiny web interface for both Windows and Linux. He went off and set up an Ubuntu box and hours later we had our RabbitMQ setup. I started to play with MQTT-SPY.

Well, it didn’t take long for the varnish to come off. One of the things you might do when you have a home control system is have some gadgets saving power by sleeping and then just waking up occasionally to do the job. MQTT has a couple of ways to help you – QOS=2 and RETAINED status. You’ll see this in MQTT-SPY if you select the slightly more complex screen option – Window – Perspectives – Detailed option.

The idea is simple – someone sends a message – someone else isn’t listening right now – but when they do – they are guaranteed to get that old message as against the default status where they don’t get old messages.

Guess what – RabbitMQ doesn’t support either of these options. Despite the very pretty interface, that was the end of that for me. So what next?

Well, didn’t I discover this little thing..

https://www14.software.ibm.com/webapp/iwm/web/preLogin.do?source=AW-0U9

MQTT in actionApparently “Really Small Message Server” will run in as little as 200k and it’s a available in a variety of platforms. Anyway – the installation could not have been simpler. I unzipped the zip file into a directory – navigated to the Windows directory within and ran the program. Done, dusted.  Of course, no password etc.. but they’ve taken a different approach to Mosquitto – you start with no config and no passwords and add them in as you need them – the documentation is VERY clear.  I made a simple config text file and put in the name of a passwords file. I made a simple passwords file with a couple of names and passwords in it – and ran the program – it worked immediately.

I hooked MQTT-SPY into it – and tried the various alternatives – QOS 0-2, retain etc.. all worked perfectly.

That was last night – this morning the little DOS box is still sitting there running Really Small Message Server. I’ve no idea how fast it can work but put it this way, I can vibrate my finger as fast as the next guy and managed to get 100 button presses in around 2 seconds – the program FEELS as if it will handle MANY times that speed. That’s accepting a message and firing it back out. Not a small message either as you’ll see.

MQTT

In the model above you see how this works – generally you are sending a topic – which generally will be in this format device/topic  so for example “03434/lightcontrol” and the data might be as simple as “on” or “off” – it is however possible to send a very long string and I send things like “red=255;green=5;blue=25”  – I’ve had the ESP8266 wrap messages in {} before sending out the serial line so they come out as this {red=255;green=5;blue=25} – the Arduino easily picks up the starting character (I could make it more complex) and stores the string till the end character (or out of buffer space), rips the commands apart and goes off to sort my serial LED lighting out.  In the future I’ll have the ESP8266 do this directly – GPIO2 could directly control a set of serial LEDs.

One could imagine putting Really Small Message Broker on a Raspberry Pi though I did read that Mosquitto has a little difficulty – high speed messages coming in tend to get taken in and not until there is a gap do the responses go out – or so I heard. I’m thinking an old laptop might be a better bet. Having said that one of my PCs stays on 24-7 – I also have a Synology Diskstation – it would be REALLY nice if I could convince Really Small Message Server to run on the Diskstation as that thing runs day-in-day-out without a hiccup.

tmp3157

I modified the MQTT software on the ESP8266 just a little. In the default demo, any incoming messages display on the serial port along with status information. I’ve wrapped the messages in {} so that my Arduino can easily pull them out.  The next job will be to allow the Arduino to send via the serial line, messages and subscribes etc.. but I’m waiting for a fix to the ESP8266 library before I do that…. https://github.com/tuanpmt/esp_mqtt – you’ll see we’ve left issues. One resolved, what both my friend Aidan and I would both like to see is a development of this with a web-based setup so you don’t have to recompile every time – it has promise.

Advertisements

26 thoughts on “A World of MQTT on ESP8266

  1. It occurs to me that you could install a mqtt client plugin in your chrome browser. I tried the chrome mqttlens plugin in my PC and Devian server and it works well.

  2. Happy New Year, Peter, and thanks for yet another stellar write-up of your progress.

    I’ve always anticipated that the most practical home/office/factory IoT system would consist of simple sensors and interfaces talking (only) to a central server/controller over a private network, which itself might then share SOME info or take SOME commands from the Internet. For my own playing around I was anticipating some hand-rolled HTTP stuff, because that’s my area… but then you clued me into MQTT… and it was immediately clear that this protocol fits the requirements perfectly, so that’s my new direction too.

    On my home network I have an old (ex-Win98) PC running as a headless Ubuntu server, so it took like 4 minutes to install mosquitto on that, and I’ve put Mqtt-Spy on my laptop, and now I’m just working through learning more about the protocol and configurations. As you know the eclipse-based build system on Windows works really well, and I’ve added a batch file in every firmware output directory which invokes esptool.py, so flashing is also a doddle, as you might say 😉

    Cheers.

  3. I have rsmb crosscompiled for my Synology DS212+, that is using a Marvell Arm processor and it works just fine. What Synology version do you have?

  4. Hi Peter,
    excellent write-up, and I have to say right on time! I was searching around for some message broker/middleware I could use for my ESP8266 modules, and MQTT looks really cool and promising. Probably exactly what I need! I will see how I can tap into MQTT in the next days/weeks (I need to finish two other projects running in parallel). Thanks again! And continue to let us know your progress 🙂

    • Thanks for that Erich – do let us know how you get on – still waiting for a fix to appear from the fellow who’s written the MQTT driver for ESP8266 – the last but one version worked well then something went wrong – but I’m sure that’ll be here in days rather than weeks. I’ve already done Windows compilations with the previous version and it looks pretty good.

    • Yes and no – it’s being developed now as I understand it – anyway the point being that it’s easier to use than Mosquitto – or rather, the documentation is easier to use. I did manage to get it to run as a service under Windows 7 but I can’t get either of them to run as a service in Windows 8 right now.

      • Per my comment on your other post, for clarity for readers here:

        Once upon a time there was a binary-only broker from IBM called RSMB.
        Then Roger Light wrote his own clean room open source broker designed to be compatible: Mosquitto.

        Years passed… literally…! 🙂

        IBM open sourced their MQTT client code (Eclipse Paho); Mosquitto became an Eclipse project; IBM open sourced RSMB and added the code to Eclipse Mosquitto as an alternative implementation / with a view to a merger of the code bases in future.

  5. Hi I’m trying to get this stuff working, but there are so many knobs to turn 🙂 Firstly, did you change the port in the ESP8266 code? and than for the broker and the mqtt-spy too?

    When I connect with Coolterm to my ESP8266 wit mqtt firmware it says this:
    TCP:Reconect to: 192.168.1.101:8443
    TCP: Reconnect to 192.168.1.101:8443
    TCP:Reconect to: 192.168.1.101:8443
    TCP: Reconnect to 192.168.1.101:8443
    TCP:Reconect to: 192.168.1.101:8443
    TCP: Reconnect to 192.168.1.101:8443

      • Thanks for the reply, I really am missing something here.. I have changed the port to 1883 (I’ve checked) and for testing I set the broker to test.mosquitto.org, but it seems like the module is not connecting. this is the response:

        TCP:Reconect to: test.mosquitto.org:1883
        DNS: found ip 85.119.83.194
        TCP: connecting…
        client handshake start.
        TCP: Reconnect to test.mosquitto.org:1883
        TCP: Connect to domain test.mosquitto.org:1883
        TCP:Reconect to: test.mosquitto.org:1883
        DNS: found ip 85.119.83.194
        TCP: connecting…
        client handshake start.

  6. Oh I think I know what that is – I got that when I didn’t change the constant (last item on the connect line) from secure to non-secure – I think I mentioned that somewhere. Tha’ts exactly what I got. Sorry I’m not in front of the code.

    • Thanks! you mean this line: #define CLIENT_SSL_ENABLE?
      I uncommented it, but still the same problem, it seems to connect, because I get this:

      .STATION_IDLE
      ip:192.168.1.121,mask:255.255.255.0,gw:192.168.1.1
      TCP: Connect to domain test.mosquitto.org:1883
      DNS: found ip 85.119.83.194
      TCP: connecting…
      client handshake start.
      TCP: Reconnect to test.mosquitto.org:1883
      TCP: Connect to domain test.mosquitto.org:1883
      TCP:Reconect to: test.mosquitto.org:1883

  7. There is a package available for Mosquito on http://synocommunuty.com, that’s what I use. Just install, and run. Default configuration works fine.

    I don’t know why you worry about SSL on a private network, say someone hacks into your local network, I’m pretty sure you’ll have more important problems to worry about. Or maybe you have a *not so private* network?

    As for using the Arduino to subscribe and publish there’s an MQTT library available called PubSubClient that is compatible with the library I developed for the ESP8266 at https://github.com/Diaoul/arduino-ESP8266
    Just replace the EthernetClient with the ESP8266Client and it’s done.
    This is as reliable as it can be given the way AT works and its lack of software flow control. I believe those issues will be addressed by espresssif in the future so I wouldn’t worry about that.

    • Hi – couple of things..

      1. Your link to synocommunity is dead.
      2. Not internal – gaping great hole to talk to the MQQT broker from my phone from anywhere in the world.. in the past I used TCP/IP – socket – which really WAS open to abuse but hopefully the MQTT version will be more secure – especially if I use HTTPS: As a matter of interest can the PUBSUBCLIENT code hook to a secure server? Not used that for AGES

      Pete

    • Somewhat confused (but that’s quite easy for me – installed the synocommunity – installed Mosquitto – it runs – erm, how do you set up a username and password on it? In the windows command line version it’s easy to add a password file – no idea how to do this on the Synology version. Any idea?

      • Most than certain that we need to ssh to it and find the configuration file somewhere under /volume1 and changing the configuration there.

  8. All running. Got MOSQUITTO running on the diskstation, Got MQTT-SPY autostarting a script- would be REALLY good to have the ability to have the Diskstation publish stuff – specifically publish a ./TIME/ message regularly so all my little boards will know what the time is – hey why waste code when you can have this!!! I noted Domoticz is at SynoCommunity – yes, that looks great but I’ve put for testing, Domoticz on the PC – it runs, it looks pretty but I need it now to talk MQTT – I would hope one could use the likes of MQTT-SPY to test communications with Domoticz – to see if it’s really any good, before worrying about how to get that code running in a specialist environment like the Diskstation (the Windows version would also be more useful to other readers in here,some of whom will already have MQTT running on their ESP8266s and might be itching to use this in a real application.. Ideas welcome.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s