ESP8266–a Way Forward

Right, it’s becoming pretty clear how this is going to develop for some of us… I’m not interested in joining someone else’s circus when it comes to home control – I want to do my own thing… but I want inexpensive mechanisms to help me do that and it’s becoming clear that MQTT as a communications mechanism is going to be part of that. It’s also clear that the ESP8266-type boards are also part of that.

So here’s where I think we are right now – but firstly I need to explain MQTT for absolute beginners before they run away:

I’ve been messing with home control over 2 years now, I wasted a year of my life on those AWFUL NRF24L01 boards which can’t even manage a single stone cottage wall and won’t send and receive at the same time. Then I moved onto better radios and even did some work with a friend on the Atmel radio processor boards – and then all of that seemed to change – the ESP8266 WIFI boards came about. Why were they better? Well first off they could use your existing WIFI infrastructure, they had excellent range, they were CHEAP – and by that let me make my position clear – I think spending £30 to turn a 50p lamp on and off is STUPID… it needs to be a LOT cheaper – like “a few quid”. When it comes to complexity – a little WIFI board on it’s own is sometimes all we should need (temperature sensor), or for a more complex item that feeding, say, an Arduino. In the past the problem has been that by the time you add Ethernet libraries and radio libraries to the Arduino you end up with almost no memory left.. and hence my use of the more powerful 1284 chips on custom boards – but that’s not REALLY the answer.. Micro-Arduino boards from China at £2 is REALLY the answer.  So a WIFI board+micro-Arduino for £4 or so – that’ll do.

The ESP-01 board is a perfectly usable board which sadly has a couple of things missing – the link for SLEEPING being one of them – the chip pins are way too small for most of us to solder and so it looks like the slightly more expensive ESP-12 might be an answer – it also has lots of FCC and other stickers to keep those people who care about such things happy.

But how do they all talk together? It seemed at first that the ESP8266 would do it all then reality set in – you can’t be a client, talking TCP/IP to a phone – AND be a web client, getting the time etc. at the SAME TIME – OH DEAR, the dream was starting to come apart. And then along came MQTT.  It’s REALLY simple… you have a machine or service somewhere called an MQTT broker and all it does is this.. everyone logs onto it with a unique name, lets say we call our devices peter and paul. Peter and paul can receive messages from the “broker” and send message – to anyone. So Peter can sent a direct message to paul. Problem solved. But it gets better.

The TOPIC system lets peter only subscribe to messages he’s interested in.  So, if Peter was an external temperature sensor he might only be interested in messages requesting the temperature.  If Paul however is a thermostat control, he might be interested in knowing what Peter’s temperature is – and what the time is . He might also want to know if he’s on PEAK time or not, or what the current requested temperature is. If this is not exciting I’m doing a piss poor job of explanation as it’s got me excited.  So whatever this protocol is it needs to be available on PCs and Linux etc., it also needs to be available on ESP8266 for those projects that use it – and on Arduino for those projects with a hardwired Ethernet connection. Well, as of around now – IT IS!

MQTT is now available on the ESP8266 but BEWARE – the author himself says this is for messing with and not yet for production – so don’t get TOO carried away – a new release came out yesterday with SSL. https://github.com/tuanpmt/esp_mqtt – my own experience is – it WORKS. I’ve had a little light sitting there dangling from a shelf, listening to the broker and responding – and it just keeps on ticking – loss of power no problem – it picks up where it left of.

But what if the little device is not on all the time – won’t it miss messages? Well no because there are 3 types of message (it’s important this is implemented in whatever you use. Type 0 message means send it and take pot luck if anyone gets it-  type 1 means they are guaranteed to get the message – but they might get it more than once. Type 2 means they will get the message for sure – once only.  This is nothing more for you to worry about than putting a number at the end of a function – 0,1 or 2.

So, let’s backtrack – that time thing. Right now I have an Arduino which calls a web page I wrote to get hte time because I don’t find NTP time servers that reliable (they get a lot of use) and they don’t send me lighting up times etc. I wrote my own PHP page sitting on a public server so I could get what I want, when I want.  And that was fine, it’s in the blog and some of you have used it… but there’s a BETTER way. Given an MQTT server and the common ability of web servers using CPANEL to allow “cron jobs” – WAY more complicated sounding than it is… imagine if I could have the ISP call that page every 15 minutes. Imagine now that the page instead of returning the time and light up times, sends that info to the BROKER!!! (As it will from sometime this afternoon until eternity). NOW the lonely thermostat ESP8266 in the kitchen doesn’t need a whole new protocol to learn – it just subscribes to that message and every 15 minutes it gets the time – SIMPLES!! But by now your ears are probably pricking.. hang on.. if I can do that then… yes, the humble web page now not only can send off the time, but a load of stuff like “lamp 4 in the living room – it’s time to turn on”. Instead of having to write a whole load of shite in a little processor you can have a password protected page on a website with TONS of terms and conditions to setup and you can update them on any old tablet and… well, I’m sure you’re writing code already.

What’s wrong with the above..

The little ESP8266 powers up for the first time, has no idea what your router is called.  So what we need is the MQTT software on the board, COMPLETE with the simplest page…. what is the router called – what is the password and what’s the address, name and password of the MQTT broker. As the ESP8266 can be it’s own access point this could all be done by a mobile phone which talks to the unit which ONLY is an access point if you pull a lead down to ground – or better – only when it actually has no information stored or the info stored does not work. Anyone up for that.

Lua allows people with no engineering degree to interact with the ESP8266 boards – and it’s very nice and there’s one fellow suggesting a way to merge MQTT and LUA – that would be great – but again BEWARE – the Lua installation WILL let you down if you get too ambitious – the implementation is unforgiving when you make mistakes and it EATS RAM.

If we could combine ALL of the above – a Lua interpreter that has it’s own web setup page – and the MQTT client software and maybe a range of temperature sensor libraries (the DHT22 is good – reliable, accurate and gives both temperature and humidity)….

BUT we won’t always be working with the ESP8266 alone – we might want it to be part of a package with an Arduino.. SO –  the unit when it receives messages should always put them out onto the serial line – maybe wrapped in some unique combination of characters – like << and >>or similar so it’s a doddle for the Arduino to pull out the message from any other crap (like the ESPs start-up 78kbaud crap we can’t seem to stop) with the absolute minimum or parsing… so for example if we have a topic peter/stuff and a message “turn something on” – maybe <<peter/stuff:turn something on>>   – see what I mean – look for “<<” – store until the colon in one buffer then till >> in another – and you have 2 buffers – topic and message – SIMPLES!!  But then there’s the slightly more complicated bit – they Arduino would need to send into the serial port two things…. 1. a topic/message in the same format – but ALSO commands to subscribe or unsubscribe to topics…   I honestly think at this point that this is ALL we need from the ESP8266… that of course and the whole thing being in a Windows- Eclipse compatible GITHUB package we can drop in – open up ONE web page in which we put our own special bits – using the Espressif SDK manual which they seem to be rather reticent to part with.

MOSQUITTO is a free MQTT broker (or server if you like) and it’s fine – but it’s all command line and pretty gruesome for beginners to understand. It can run on a PC under Windows… it’s ACTUALLY not that difficult once you’ve used it but messing with that config file is like a trip back into the 20th century – someone needs to make a nice sparkly Windows screen with lots of tick boxes.  When it comes to clients on the PC I’m in love with the free MQTT-SPY – it’s just lovely.  As for Android – really – MyMQTT is awful – if anyone wants to re-write MQTT Spy for Android they would be on my Christmas card list for years.

So there you are – I’m playing with Mosquitto and Rabbit-MQ which looks really nice but I have to say it did collapse when I tried to send a level 2 message out… might just be me not understanding things yet.

From here I’m ready to start coding – there’s enough above to form the basis of a very large and complicated control system –  I’m THAT far off having an internet controlled SAD light for the wife… it’s sitting here waiting for commands but I need something on the phone with nice round, shiny buttons and flashing lights that looks the part – not sure Basic 4 Android is going to do that for me but we’ll give it a whirl.

Let me know if this didn’t make sense – because if it didn’t you’re missing out on lots of potential fun.

Regards

 

Peter Scargill

p.s. The easiest way to get to grips with this MQTT thing – get MOSQUITTO – install it  – ignore usernames and passwords for now.  Get MQTT Spy – tell it about the IP address of the machine with MOSQUITTO (can be same machine) and PLAY… subscribe to arbitrarily named topics – send messages to those topics – within minutes it will all become clear. 

p.p.s. If you’re a Facebook fan and want to read about this stuff ONLY for Windows users who like pretty interfaces like me – head over to www.facebook.com/esp8266wifi