A World of MQTT on ESP8266

From no-where, I’ve been immersed in the world of MQTT for the past day or two. If you’ve read my previous posts you’ll know I’ve been working on home control for years, mainly as a hobby and so mainly on a budget. I’ve tried several systems out there and eventually settled on doing my own thing because many of the packages have been developed so far then stopped or they’re very complicated or expensive etc..

My earlier posts will show you that I’m walking around with a mobile phone with fairly decent control over 3 properties right now, so it’s not that I don’t have solutions – but the controls could be more responsive, I’ve had a hell of a time with the NRF24L01 radios which I plan soon to put up against a wall and shoot and I was in the process of swapping over to some 433Mhz units when out of the blue the ESP8266 modules appeared and changed all that.

Anyway there is one guy developing a network of these things – early days yet and when I re-discovered an MQTT library was available for the ESP8266, it seemed the natural way to do the job (don’t rush – the version out there yesterday has developed a big fault – give it a day or so) – each unit, be it an ESP-01 on it’s own with a sensor – or attached to an Arduino or something more powerful, would be able to send to and received messages from any other unit – marvellous.

The “broker” for messages is the package that accepts messages from devices (or PCs or whatever) and makes them available to other devices. Loads of advice from people to use “free” services – but after wasting many hours – they all come with a catch. Limits on use, lack of guaranteed they’ll actually be up and running or in the case of one – your messages are open for everyone to see. NO THANK YOU.

The obvious choice was a local copy of MOSQUITTO – the free package designed, apparently by Roger Light. A simple DOS application which can also act as a service. Marvellous. Now I needed to put the code into the ESP8266 – that worked – marvellous. I even developed my own little client application (well, I tarted up a demo) on Basic 4 Android (B4A). Now I needed some PC client to test the messages… not so marvellous. The command line tools provided with Mosquitto are, well, command line tools – I may be 60 but I’m still part of the pretty-coloured-buttons generation – I really would rather have as little to do with black boxes as possible.  At least half of the links I tried on the web went to projects that haven’t started yet, or old ones that were dead – or just no-where. I finally settled on MQTT-SPY and I have to say.. it’s lovely both in operation and visually. MQTT messages are really trivial comprising a topic – and some content. I set up Mosquitto, subscribed to an arbitrary topic – and sent a message to it. LO AND BEHOLD it appeared at the bottom of MQTT-SPY.

I realised at this point that the messages were utterly non-secure and started to read the documentation for Mosquitto. After a while as I started to ponder cutting my wrists I came to the conclusion that this package was a TAD over the top for my requirements – I got password protection working and tested it out – very nice – but then two of us were using this and in both cases the package failed once.  At that point my friend who was helping me got very excited as he’d discovered RabbitMQ – a VERY pretty alternative with a nice shiny web interface for both Windows and Linux. He went off and set up an Ubuntu box and hours later we had our RabbitMQ setup. I started to play with MQTT-SPY.

Well, it didn’t take long for the varnish to come off. One of the things you might do when you have a home control system is have some gadgets saving power by sleeping and then just waking up occasionally to do the job. MQTT has a couple of ways to help you – QOS=2 and RETAINED status. You’ll see this in MQTT-SPY if you select the slightly more complex screen option – Window – Perspectives – Detailed option.

The idea is simple – someone sends a message – someone else isn’t listening right now – but when they do – they are guaranteed to get that old message as against the default status where they don’t get old messages.

Guess what – RabbitMQ doesn’t support either of these options. Despite the very pretty interface, that was the end of that for me. So what next?

Well, didn’t I discover this little thing..

https://www14.software.ibm.com/webapp/iwm/web/preLogin.do?source=AW-0U9

Apparently “Really Small Message Server” will run in as little as 200k and it’s a available in a variety of platforms. Anyway – the installation could not have been simpler. I unzipped the zip file into a directory – navigated to the Windows directory within and ran the program. Done, dusted.  Of course, no password etc.. but they’ve taken a different approach to Mosquitto – you start with no config and no passwords and add them in as you need them – the documentation is VERY clear.  I made a simple config text file and put in the name of a passwords file. I made a simple passwords file with a couple of names and passwords in it – and ran the program – it worked immediately.

I hooked MQTT-SPY into it – and tried the various alternatives – QOS 0-2, retain etc.. all worked perfectly.

That was last night – this morning the little DOS box is still sitting there running Really Small Message Server. I’ve no idea how fast it can work but put it this way, I can vibrate my finger as fast as the next guy and managed to get 100 button presses in around 2 seconds – the program FEELS as if it will handle MANY times that speed. That’s accepting a message and firing it back out. Not a small message either as you’ll see.

In the model above you see how this works – generally you are sending a topic – which generally will be in this format device/topic  so for example “03434/lightcontrol” and the data might be as simple as “on” or “off” – it is however possible to send a very long string and I send things like “red=255;green=5;blue=25”  – I’ve had the ESP8266 wrap messages in {} before sending out the serial line so they come out as this {red=255;green=5;blue=25} – the Arduino easily picks up the starting character (I could make it more complex) and stores the string till the end character (or out of buffer space), rips the commands apart and goes off to sort my serial LED lighting out.  In the future I’ll have the ESP8266 do this directly – GPIO2 could directly control a set of serial LEDs.

One could imagine putting Really Small Message Broker on a Raspberry Pi though I did read that Mosquitto has a little difficulty – high speed messages coming in tend to get taken in and not until there is a gap do the responses go out – or so I heard. I’m thinking an old laptop might be a better bet. Having said that one of my PCs stays on 24-7 – I also have a Synology Diskstation – it would be REALLY nice if I could convince Really Small Message Server to run on the Diskstation as that thing runs day-in-day-out without a hiccup.

I modified the MQTT software on the ESP8266 just a little. In the default demo, any incoming messages display on the serial port along with status information. I’ve wrapped the messages in {} so that my Arduino can easily pull them out.  The next job will be to allow the Arduino to send via the serial line, messages and subscribes etc.. but I’m waiting for a fix to the ESP8266 library before I do that…. https://github.com/tuanpmt/esp_mqtt – you’ll see we’ve left issues. One resolved, what both my friend Aidan and I would both like to see is a development of this with a web-based setup so you don’t have to recompile every time – it has promise.