ESP8266 and MQTT A Marriage Made in Heaven?

It’s been a very busy day – and a very frustrating day. I’ve had to make a start at figuring out how MAKEFILES work because of a fellow who implemented MQTT for ESP8266 so I could seamlessly compile his code and blow ESP-01 chips with the Eclipse environment in Windows.

It all started a couple of days ago when I utterly accidentally stumbled on a project to implement MQTT onto an ESP8266. I don’t know why I went for this but I’d just gotten to grips with the Eclipse setup so I was now a fully fledged Windows developer of ESP8266 software.. by that I mean I’d figured out how to alter code to get a light to flash on and off in the AT command software.

I remember playing with MQTT a while ago and I mentioned it in here or on the ESP8266WIFI Facebook page. Anyway I sat down with a large glass of wine and started to tinker. Essentially, MQTT is a lightweight protocol for sending messages around a network – apparently Facebook Messenger uses it.  You have a “broker” – call it a server if you are a techie… so this is free software like for example MOSQUITTO which you can run hosted, on your PC or on a Raspberry Pi. you can make an account with it – which might be open, have a simple password or have something far more complicated.

One of my fears of controlling the house remotely has always been security and I’ve made some primitive passwording which is really not worth a light.. so this caught my interest. My other issue as regular readers will know was that I was getting utterly hacked off with the ESP8266 as it can’t really do server and client at exactly the same time – so you would have trouble making a network of them – you CAN do it and there’s a slow moving project on the subject… but right now, not a viable solution – that might change.  The instant I saw MQTT my heart skipped a beat as I thought… can’t one device send a message to another?

I put IBM’s free MOSQUITTO on my PC (powers up with no passwords etc, doddle to use even if the command line environment makes some people feel physically sick – including me). IT uses a single port which you can change so it’s easy to make available outside via a little port mapping on your router.

I blew the code for this onto one of my ESP8266 boards – and left that connected to a serial monitor. Sure enough, it connected first time to MOSQUITTO. It SUBSCRIBED to a couple of messages (utterly arbitrary and they don’t need to exist at the time of subscription).

I grabbed a free but not very good program for Android called MyMQTT. IT’s not seen an update in AGES but I remembered it worked ok. I told it about my MOSQUITTO setup (ie the IP address of my computer) and that was that. I subscribed on the mobile APP to the same “topics” my ESP8266 had. I sent a message to that topic… sure enough instantly on the phone, the topic I’d replied to came back with a message – the same one I’d sent out… simultaneously the same message came into the ESP8266. We’re talking INSTANT here – no visible delays.

Immediately I grasped the possibilities. I modified the MQTT code (in the new sparkly ECLIPSE environment) on the ESP8266 to check the content of one of the topics it had subscribed to… a quick STRSTR on the zero-terminated incoming string… I made 2 comparisons – one for a message “GPIO0 ON” and another for “GPIO0 OFF”. I borrowed the heater and port info from a BLINKY sketch (all of 3 lines of code for everything)… and compiled away. I sat in the corner of the room and gingerly entered “GPIO ON” into my mobile phone app… and… BINGO the light came on.

Step 2. When the light comes on – SEND a message to the same topic but a different device number – I should say the author engineered auto device numbers based I think on the mac codes – they’re all unique…   /0033342342/data as it happens – I sent a hello.

Sat back with the phone, put my GPIO message in again, the light came on and IMMEDIATELY the message appeared on the SECOND ESP8266 as expected.

So with the minor inconvenience of a piece of “broker” software minding it’s own business on a minimal PC, a Raspberry Pi or some external server (your choice) the ESP8266 boards can now talk to each other without another processor. Not only that, I reasoned but if the server machine also had a client, it should publish a “TIME” message every 10 minutes or so…  the boards would IF APPROPRIATE be programmed to subscribe to this channel and LO  – ESP8266 boards which can turn things on and off – and also know what time it is. But what of the power of an Arduino – what about all those extra pins? Another project implements I2c on the ESP8266 and when my skills are ready I’m adding THAT into a mix – which means any number of peripherals to do things or return info using MQTT.

It’s a grand total of £40 or so investment to get yourself into the Android App trivial programming brigade using B4A.  Mine of out of date, out of warranty and so I’ll ahve to start again. For £10 a guy has written a library for MQTT and this afternoon on a friend’s PC – someone who has the full B4A, we wrote a simple messenger program that will send and receive MQTT messages. The results – again absolutely instant message passing back and forth.

Imagine – you have a button on the phone – you press it and it does two things.. it turns a little indicator called kitchen_lamp to grey then sends a message to /kitchen/lamp which it does not subscribe to – the message is ON… the kitchen lamp ESP8266 box which features nothing more than an ESP-01, a 3v mains PSU and a relay, gets the message and turns off GPIO0 which turns on the relay (there’s a good reason to do it backwards like that). It then sends a message back to another topic which it does not subscribe to called /kitchenresponse/lamp. THe phone however DOES respond to that – and when it sees an incoming message to that topic, it turns that same indicator green or red depending on the message.  So the lamp is off – you press the on button and your indicator on the phone almost immediately turns green so you know for a FACT that the kitchen light came on.

You might take that a stage further – because you’ve written the client on your PC to send messages like TIME, the kitchen lamp knows what time it is because it also subscribes to TIME.  if no-one turns that lamp off, at midnight it turns it off itself.

Multiply this by dozens of similar gadgets, some DO stuff like turning things on and off, others merely sit there, read the temperature and report it back – either every now and then – or on demand… same technique..

An old Android tablet sits on the wall, it has a program you wrote which subscribes to as many or little of the topics as you want and reports in pretty graphical format the various statuses-  it might even have a large round touch control reminiscent of a a certain apple-esque temperature controller…. and everyone talks to each other securely over WIFI..

BUT you might say, I really want to use Arduinos – ok you have the little ESP8266 do all the hard work of handling messages incoming messages it sends out of the serial port and if you want to send a message that could come in through the serial port… so now you have an Arduino that has to do nothing but take in serial messages and send them out – polling is easy – and you can of course wrap messages in any control character or sequence you want.

But hang on – isn’t Arduino serial interrupt driven? YOU BET – which means if you go in and tinker with that you could have that look for the starting sequence, start storing bytes until the ending sequence occurs, copy that temporary buffer to another, set a flag to say there is a massage – and simultaneously go back to looking for the next message – all using existing resources that are used in the serial handling anyway.  The Arduino ends up with a few bytes less than a full pack and can now receive serial messages and act on them with virtually no extra overhead which means the simple 328 has LOTS of room to DO stuff.

That enough for now? It certainly has me excited but I’ve had enough for one day… I need to turn most of this from thought processes to reality. The MQTT project is apparently in it’s early stages but I can assure you it sends messages with password protection as reliably as it receives them. Even if the designer does nothing else you have this at your disposal – links in earlier entries in this blog. Go get ‘em.

Now if anyone’s up for some collaboration I’d much rather someone else write the Android APP – a decent replacement for the promising but dead in the water myMQTT..  All in good time – good night all.

Advertisements

24 thoughts on “ESP8266 and MQTT A Marriage Made in Heaven?

  1. Hi Peter –

    It may make your head spin, but the OpenHab (http://www.openhab.org/) and OpenRemote (http://www.openremote.org/display/HOME/OpenRemote) folks have developed a Java server (run it on a RPi, NAS box, your PC, whatever) that speaks MQTT and already has their associated Android control applications:

    https://play.google.com/store/apps/details?id=org.openhab.habdroid
    https://play.google.com/store/apps/details?id=org.openremote.android.console

    Of course the hard part is setting up all the configuration files etc… that need to be tweaked and tailored for your home. But still, if you can stomach the effort, a lot of the heavy lifting has already been done for you. Probably better to build on top of one of these projects than roll your own. I’m looking to play with these when I get the chance. Just got my first ESP-12 a few days ago and wired it up.Still busy building the SDK and tool-chain. Should be fun to tinker with it provided work doesn’t get in the way 😉

    Have fun . . . been enjoying your blog . . .

    Glenn

  2. Hi Sameh

    Ok, well first off I’m not that familiar with NODEJS but I tried a few online attempts – and they don’t work because (am I right?) the standard build of MOSQUITTO has not been compiled with websockets (is that the right term?).

    • Hi Pete,

      It should work without problems, all what you need is to install the MOSQUITTO broker server, and then install nodejs, and install MQTT dependencies with one command line ” npm install mqtt ”

      Then you are ready to go and can test with something like:

      var mqtt = require(‘../..’)
      , host = ‘192.168.1.100’ // or localhost
      , client = mqtt.createClient();
      // or , client = mqtt.createClient(1883, host, {keepalive: 10000});

      client.subscribe(‘presence’);
      client.publish(‘presence’, ‘bin hier’);
      client.on(‘message’, function (topic, message) {
      console.log(message);
      });
      client.end();

  3. I’ve tried openhab, and homegenie in the past and both are fairly decent projects for messing with mqtt, but in my humble opinion, pidome is the most polished of all the home automation apps, and certainly one to follow. It may be in alpha currently, but it is absolutely rock solid with a couple of very nice clients (android and PC).

    Pidome runs on a rapsberry pi, has mqtt built in and is really simple to use. The author is very responsive to emails. http://www.pidome.org, If you’ve got a pi lying around, it certainly worth having a play with to see if it’s up your street 🙂

    Glen.

    • Thanks Glen – I bought a Pi when they first came out – loaded Linux, realised it was as slow as a dog and put it on Ebay. Some day I may re-visit. I saw reference to another Android client but it would not work with Mosquitto. I think for now I want to stick with plain MQTT… the B4A tools look great and yesterday we had a client with publish and subscribe running – sadly today on my PC here the project is griping about a missing file. I think I’ll have to buy my own B4A as that’s the only way to get support or be able to download their various projects. Happy with the PC clients I’ve discovered today but Android really does need more as MYMQTT hasn’t moved on in months. The same guy who did the library for B4A has some pretty damned impressive buttons available for free by the looks of it. I’ve been using NETIO but that too hasn’t changed for many months – time to move on.

      • Slow is relative 😉 They are ideal for a low power control centre. I use them all over the place, media centres for all the tv’s, phone control (asterisk server), network music players, NTP server, the list goes on and on :). They do make for an ideal MQTT server as well.

        I don’t know linux too well, just the basics, and enough to follow online instructions.

  4. I’m also checking what I could do with esp8266 and the MQTT and CoAP protocols, being more inclined to the CoAP protocol do to fact it doesn’t require an open connection at all times. But it comes at a price that by using UDP instead of TCP, some application logic for retries and acknowledges… The fact is that the esp8266 is not so that constrained at all, because it already has a TCP stack… so CoAP might not make sense. I think only the network topology might make the decision between one or another protocol.
    Regarding the Android apps, check out also Kivy (www.kivy.org) that is free, uses python for programming (rather simple) and allows multi platform deployments and there is I think some MQTT clients out ther.

  5. Hi Peter

    It is good to hear about the progress you are making.

    By B4A I assume you are referring to Basic4Android.

    Looking at their web site I see that this costs $59.

    Surely the implied ethos of the ESP8266 development community is to produce a minimal cost IoT node by using freely available, and free, development tools. Hence B4A is not a preferred path.

    Although B4A may have some ‘tasty’ bits and bobs, is there a suitable free Android Basic?

    Using Google to find an alternative to B4A I found RFO Basic!, which is a dialect of Dartmouth Basic targeted at the Android OS.

    As I turned my back on Basic over 30 years ago, I have no way of knowing if this is a useful implementation.

    Can your contributors identify a suitable Android Basic?

  6. Hi

    Yes it’s around $59. It’s a tool – and it has (though I’m having difficulty) an MQTT library. I’m not sure I follow your implied community ethos.. I don’t feel bound to move to free or expensive but to use what does the job for me and hope that my notes help others. Somehow I don’t think Espressif are in this for fun. I want my mobile apps to look professional and if I have to pay a little money for that, so be it. Of course free is good but I think that other Basic you refer to is free for a reason. On the other hand, elsewhere in here someone is referring to KIVY and having just seen a demo I quite like the look of it. I’m going to give it a quick investigation. It’s all down to support – B4A produces pretty nice output and there are some swish libraries for it. I really don’t want to develop my own dials and knobs and lights.. I’ll let you know how I get on with Kivy, meanwhile the designer of the MQTT library for B4A has just offered a little help.. his library costs a tenner to remove a message ut if it works reliably, reconnecting when it fails etc, then it might be worth it.

  7. Sorry. Had to answer here. Couldn’t answer to your question above, below it.

    For running/having Kivy all that is needed is Python 2 ( not 3) and Kivy installation on the PC. There are instructions for multiple platforms for installing Kivy namely Windows ( I’m a Linux user… 🙂 )
    The code that is developed works on the PC and on Mobile without modification. For Android the app can be lauched from the Kivy Launcher or compiled into standalone APK using a build system that compiles python into native Android (Dalvik Java?, Andorid NDK?).

    I think Kivy is worth a look. It’s free and that means that everyone can check out things freely. Just my opinion 🙂

    While NodeMcu is fine with the Lua language, MicroPython on the esp8266 would mean that we can do python for all stack components, except the MQTT server (no need to re-invent the wheel here), so we can leverage python knowledge from end to end.

    • Hi

      That’s probably very nice, Gilko, but WAY too vague – there’s not much air in the Northeast of England so we think slowly 🙂 ….. I know what MQTT is – I know what LUA on ESP8266 is is…(though I have yet to get that code compiling in the new Windows Eclipse environment) how do we go from there- to having a working installation of LUA with the MQTT in it???

      As for tools – if you look up the free Android myMQTT – you’ll see why I’d want my own tool – it’s AWFUL. It disconnects and doesn’t have the option to reconnect automatically, it can only work with one broker at once.. it would not take a skilled operator very long to make a much better Android phone/tablet client for us – on the PC I’m using MQTT-SPY which is just WONDERFUL – I would DEARLY love to have an equivalent on my Android tablet – and a slimmer version on the phone – not entirely sure my skills are up to it yet. Would need to work with MOSQUITTO and Rabbit-MQ….

  8. Just a little note to point out and clarify that Mosquitto is not from IBM – it is the result of years of hard work by Roger Light, who is quite independent. Actually it was originally created because IBM’s Really Small Message Broker (RSMB) was initially binary-only. More recently, Mosquitto has become a project at the Eclipse Foundation; and RSMB has been made open source, with the code available as part of the same project. Funny how these things work out! 🙂

    • Yes of course you are right on that Andy… I’ve gone with Really Small Message Broker because it is lightweight and up to now just works! Even on a rubbish connection from the train to London 🙂

  9. Hi, I have the code from tuanpm up and running, echoing back messages I publish from mqttspy, but where do you insert the code to respond to a message with, say, setting a gpio to high for message ‘on’, low for ‘off’?

    Would you be able to post the example of what you did to get that up and running? Just a snippet should be enough to get me going.

    Thanks.

    • Oh that’s easy – so presumably you are in user_main – in the function mqttDataCb responding to whatever is coming in. Assume you have set the port in your INIT CODE…

      This sets GPIO0

      GPIO_OUTPUT_SET(GPIO_ID_PIN(WSGPIO), 0); // for LEDS

      Then in mqttdataCB… here’s an example.. my feeble attempts to talk to OpenHab (which work).

      if (strcmp(topicBuf,”home/openHAB/out/Light_FF_Bath_Ceiling/command”)==0)
      {
      if (strcmp(dataBuf,”ON”)) // wrong way around
      {
      GPIO_OUTPUT_SET(LED_GPIO, 1);
      MQTT_Publish(client, “home/openHAB/state/Light_FF_Bath_Ceiling/state”,”OFF”,3, 0, 0);
      }
      else
      {
      GPIO_OUTPUT_SET(LED_GPIO, 0);
      MQTT_Publish(client, “home/openHAB/state/Light_FF_Bath_Ceiling/state”,”ON”,2, 0, 0);
      }
      }

      Have fun

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s