ESP8266–a Way Forward

Right, it’s becoming pretty clear how this is going to develop for some of us… I’m not interested in joining someone else’s circus when it comes to home control – I want to do my own thing… but I want inexpensive mechanisms to help me do that and it’s becoming clear that MQTT as a communications mechanism is going to be part of that. It’s also clear that the ESP8266-type boards are also part of that.

So here’s where I think we are right now – but firstly I need to explain MQTT for absolute beginners before they run away:

I’ve been messing with home control over 2 years now, I wasted a year of my life on those AWFUL NRF24L01 boards which can’t even manage a single stone cottage wall and won’t send and receive at the same time. Then I moved onto better radios and even did some work with a friend on the Atmel radio processor boards – and then all of that seemed to change – the ESP8266 WIFI boards came about. Why were they better? Well first off they could use your existing WIFI infrastructure, they had excellent range, they were CHEAP – and by that let me make my position clear – I think spending £30 to turn a 50p lamp on and off is STUPID… it needs to be a LOT cheaper – like “a few quid”. When it comes to complexity – a little WIFI board on it’s own is sometimes all we should need (temperature sensor), or for a more complex item that feeding, say, an Arduino. In the past the problem has been that by the time you add Ethernet libraries and radio libraries to the Arduino you end up with almost no memory left.. and hence my use of the more powerful 1284 chips on custom boards – but that’s not REALLY the answer.. Micro-Arduino boards from China at £2 is REALLY the answer.  So a WIFI board+micro-Arduino for £4 or so – that’ll do.

The ESP-01 board is a perfectly usable board which sadly has a couple of things missing – the link for SLEEPING being one of them – the chip pins are way too small for most of us to solder and so it looks like the slightly more expensive ESP-12 might be an answer – it also has lots of FCC and other stickers to keep those people who care about such things happy.

But how do they all talk together? It seemed at first that the ESP8266 would do it all then reality set in – you can’t be a client, talking TCP/IP to a phone – AND be a web client, getting the time etc. at the SAME TIME – OH DEAR, the dream was starting to come apart. And then along came MQTT.  It’s REALLY simple… you have a machine or service somewhere called an MQTT broker and all it does is this.. everyone logs onto it with a unique name, lets say we call our devices peter and paul. Peter and paul can receive messages from the “broker” and send message – to anyone. So Peter can sent a direct message to paul. Problem solved. But it gets better.

The TOPIC system lets peter only subscribe to messages he’s interested in.  So, if Peter was an external temperature sensor he might only be interested in messages requesting the temperature.  If Paul however is a thermostat control, he might be interested in knowing what Peter’s temperature is – and what the time is . He might also want to know if he’s on PEAK time or not, or what the current requested temperature is. If this is not exciting I’m doing a piss poor job of explanation as it’s got me excited.  So whatever this protocol is it needs to be available on PCs and Linux etc., it also needs to be available on ESP8266 for those projects that use it – and on Arduino for those projects with a hardwired Ethernet connection. Well, as of around now – IT IS!

MQTT is now available on the ESP8266 but BEWARE – the author himself says this is for messing with and not yet for production – so don’t get TOO carried away – a new release came out yesterday with SSL. https://github.com/tuanpmt/esp_mqtt – my own experience is – it WORKS. I’ve had a little light sitting there dangling from a shelf, listening to the broker and responding – and it just keeps on ticking – loss of power no problem – it picks up where it left of.

But what if the little device is not on all the time – won’t it miss messages? Well no because there are 3 types of message (it’s important this is implemented in whatever you use. Type 0 message means send it and take pot luck if anyone gets it-  type 1 means they are guaranteed to get the message – but they might get it more than once. Type 2 means they will get the message for sure – once only.  This is nothing more for you to worry about than putting a number at the end of a function – 0,1 or 2.

So, let’s backtrack – that time thing. Right now I have an Arduino which calls a web page I wrote to get hte time because I don’t find NTP time servers that reliable (they get a lot of use) and they don’t send me lighting up times etc. I wrote my own PHP page sitting on a public server so I could get what I want, when I want.  And that was fine, it’s in the blog and some of you have used it… but there’s a BETTER way. Given an MQTT server and the common ability of web servers using CPANEL to allow “cron jobs” – WAY more complicated sounding than it is… imagine if I could have the ISP call that page every 15 minutes. Imagine now that the page instead of returning the time and light up times, sends that info to the BROKER!!! (As it will from sometime this afternoon until eternity). NOW the lonely thermostat ESP8266 in the kitchen doesn’t need a whole new protocol to learn – it just subscribes to that message and every 15 minutes it gets the time – SIMPLES!! But by now your ears are probably pricking.. hang on.. if I can do that then… yes, the humble web page now not only can send off the time, but a load of stuff like “lamp 4 in the living room – it’s time to turn on”. Instead of having to write a whole load of shite in a little processor you can have a password protected page on a website with TONS of terms and conditions to setup and you can update them on any old tablet and… well, I’m sure you’re writing code already.

What’s wrong with the above..

The little ESP8266 powers up for the first time, has no idea what your router is called.  So what we need is the MQTT software on the board, COMPLETE with the simplest page…. what is the router called – what is the password and what’s the address, name and password of the MQTT broker. As the ESP8266 can be it’s own access point this could all be done by a mobile phone which talks to the unit which ONLY is an access point if you pull a lead down to ground – or better – only when it actually has no information stored or the info stored does not work. Anyone up for that.

Lua allows people with no engineering degree to interact with the ESP8266 boards – and it’s very nice and there’s one fellow suggesting a way to merge MQTT and LUA – that would be great – but again BEWARE – the Lua installation WILL let you down if you get too ambitious – the implementation is unforgiving when you make mistakes and it EATS RAM.

If we could combine ALL of the above – a Lua interpreter that has it’s own web setup page – and the MQTT client software and maybe a range of temperature sensor libraries (the DHT22 is good – reliable, accurate and gives both temperature and humidity)….

BUT we won’t always be working with the ESP8266 alone – we might want it to be part of a package with an Arduino.. SO –  the unit when it receives messages should always put them out onto the serial line – maybe wrapped in some unique combination of characters – like << and >>or similar so it’s a doddle for the Arduino to pull out the message from any other crap (like the ESPs start-up 78kbaud crap we can’t seem to stop) with the absolute minimum or parsing… so for example if we have a topic peter/stuff and a message “turn something on” – maybe <<peter/stuff:turn something on>>   – see what I mean – look for “<<” – store until the colon in one buffer then till >> in another – and you have 2 buffers – topic and message – SIMPLES!!  But then there’s the slightly more complicated bit – they Arduino would need to send into the serial port two things…. 1. a topic/message in the same format – but ALSO commands to subscribe or unsubscribe to topics…   I honestly think at this point that this is ALL we need from the ESP8266… that of course and the whole thing being in a Windows- Eclipse compatible GITHUB package we can drop in – open up ONE web page in which we put our own special bits – using the Espressif SDK manual which they seem to be rather reticent to part with.

MOSQUITTO is a free MQTT broker (or server if you like) and it’s fine – but it’s all command line and pretty gruesome for beginners to understand. It can run on a PC under Windows… it’s ACTUALLY not that difficult once you’ve used it but messing with that config file is like a trip back into the 20th century – someone needs to make a nice sparkly Windows screen with lots of tick boxes.  When it comes to clients on the PC I’m in love with the free MQTT-SPY – it’s just lovely.  As for Android – really – MyMQTT is awful – if anyone wants to re-write MQTT Spy for Android they would be on my Christmas card list for years.

So there you are – I’m playing with Mosquitto and Rabbit-MQ which looks really nice but I have to say it did collapse when I tried to send a level 2 message out… might just be me not understanding things yet.

From here I’m ready to start coding – there’s enough above to form the basis of a very large and complicated control system –  I’m THAT far off having an internet controlled SAD light for the wife… it’s sitting here waiting for commands but I need something on the phone with nice round, shiny buttons and flashing lights that looks the part – not sure Basic 4 Android is going to do that for me but we’ll give it a whirl.

Let me know if this didn’t make sense – because if it didn’t you’re missing out on lots of potential fun.

Regards

 

Peter Scargill

p.s. The easiest way to get to grips with this MQTT thing – get MOSQUITTO – install it  – ignore usernames and passwords for now.  Get MQTT Spy – tell it about the IP address of the machine with MOSQUITTO (can be same machine) and PLAY… subscribe to arbitrarily named topics – send messages to those topics – within minutes it will all become clear. 

p.p.s. If you’re a Facebook fan and want to read about this stuff ONLY for Windows users who like pretty interfaces like me – head over to www.facebook.com/esp8266wifi

Advertisements

18 thoughts on “ESP8266–a Way Forward

  1. Starting point – well if you’re on windows you need to get that new Eclipse environment going – until I got that I was very near to strangling my own cats!!!

    Ok, RIGHT from the start… ok get an ESP-01 – get an FTDI — that’s a USB to serial adaptor – maybe £3 from Ebay. Set it to 3v operation – make sure you get one with a link…. wire it to ground, vcc, serial in and serial out on the board, Make sure the board has that PD pin tied to power – and make a switch arrangement to be able to tie GPIO-0 to ground for programming (turn power off, GPIO-0 to ground, power on – program, remove link, power off, power on – you’re in business).

    Go to the NODEMCU site where the LUA stuff is – they have a programming tool… basically in many cases like the AT command set – you just have a couple of files you have to blow – I’ve documented it earlier in the blog. Get the latest command set, get something like that great new serial terminal (not mine) I blogged about – use the terminal to ensure the little board is hooked up to your router – and off you go – start experimenting – I learn best by DOING….

    Thanks for taking the time to write about you findings as you explore these options (I’m really enjoying your passionate writings…they help me imagine new possibilities)! I’m just starting to dabble with the ESP8266 myself and have a lot of projects in mind that could really benefit from the MQTT methodology you’ve been describing over the last few days. The problem I have, though, is that I’m WAY BEHIND in understanding these boards or how to use them efficiently…let alone flashing them with custom firmware. Do you have a starting point you’d recommend for someone like me to get up to speed? In case it helps, I’m not a developer, but can typically figure out what I need on the Arduino side…so starting with Arduino + ESP8266 may be the simplest option at first. After that, I love the idea of using just the ESP8266 with simple sensors and possibly communicating those results back to a master Arduino compatible device (like a Spark Core) to give me centralized control for my home systems remotely.

    • To clarify, I already have a few ESP-03 boards with two currently soldered up to breakout boards; making them breadboard friendly. From there I’m waiting on an a FTDI adapter to arrive and intend to start experimenting with the units by connecting them to an Arduino first…just to get more familiar with the basics. I’ll check out the other sites you mentioned and look through your prior blog listings (I only stumbled upon your site within the last week…so I have plenty to catch up on). Hopefully from there I’ll start to better understand the possibilities of using these boards directly with sensors! Thanks again and please keep the ESP8266 + MQTT research/write-ups coming.

    • “make sure you get one with a link” – what do you mean by link?

      ” and make a switch arrangement to be able to tie GPIO-0 to ground for programming (turn power off, GPIO-0 to ground, power on – program, remove link, power off, power on – you’re in business).”

      Is it possible to get a picture of said basic setup?

      I agree about the doing part. I also like to SEE…

    • This could be a dumb question as someone who lives in Britain where they are widely available if you’re daft enough to pay the markup that people try to make for merely passing on a Chinese product… erm… please treat this as an ignorant question… why can’t you inport from China? I don’t see these EVER landing in a physical store – not enough markup surely.

      • Hi Peter,

        No problem… I can order from the likes of DX or Alibaba, but this kind of thing usually takes weeks to arrive from China and up to 2 MONTHS to be released from customs.

        Since I’ll be in the US next week I thought you could point me to a supplier there.

        I see some on Amazon (they can deliver quickly to a Hotel) but there are a few different specs. Will a ESP-01 version V090 do it? I’ll mostly use Arduinos to play with the ESP8266.

        Thanks!

    • Gilson – There are currently some US ebayers selling the ESP8266, but I didn’t find any who are in California. You could probably get a US seller to ship to a hotel, but one week would be cutting it close.

      • Forget the local view – honestly – no longer relevant – you can order from any corner of the world and in come cases the best places to get innovation from will be far from where you might prefer.. benefit from that… I would LOVE to think that the best deals, the best innovation came from the home of the modern world – the UK – sadly… in this case the ball is in the Chinese court entirely….

  2. Just to say I’m very excited about this prospect. I’ve found interacting with the esp via an arduino using the AT commands too buggy and the massive number of arduino libraries too much to give up to go fully native development with this board.

    Might I suggest you get your own github repository going for this? I’m sure I and many others would be keen to have an easy to use mqtt + optionally arduino client going with a basis for android development.

  3. Really interesting post, looks promising with this MQTT stuff.

    I have been following your posts on the ESP8266 over the last few months or so, really helpful. This is all pretty new to me from a ‘electronic engineering’ / embedded programming point of view so it is nice to see some examples / info.

    Regards

    Cdr

  4. Hi Peter,

    Ok, I got it after you replied asking why didn’t I import from China, but should have asked like this next:

    “I can buy from Amazon but I’m still confused by the different specs and models available. Would a ESP-01 version V090 work for playing with Arduinos?”

    Sorry if this has already been answered…

    Thanks!

  5. Personally i have very limited time and managed to fry two 8266 – mostly because of voltage problems – don’t believe an usb/serial board that says it is “3.3v” logic.

    I am waiting for a “shield”/B.O.B. that has on board regulator and level shift. That was I can spend time being productive with software instead of letting out magic smoke.

    Wouldn’t it be stellar to have it do mesh…

    • Well, the MQTT option is not that far off a mesh. Any unit can effectively send to any other unit… Right now I have 2 test units, one sending keep alive packages to MOSQUITO on my computer here – and another doing the same with RABBITMQ server at a friends’s place in Northumberland (really just to compare the two – RabbitMQ has a really nice web interface)… and as I’ve spent the day sleeping off NYE excesses, they’ve both been performing flawlessly. I’m not up to date with email yet but as of yesterday, the version update of MQTT for Android referred to in my blog is faulty, we had to revert back to a version from a few days ago – but two of us have left messages with the author and I’ll comment in a future blog on developments. Yes, I built my own shields, did a little PCB layout and got 10 bare boards from China for around a tenner and built them up. As you say it beats blowing things up.

  6. This looks very nice. I’ve build my own home (room based) thermostats using a mesh network of nrf24l01/433/atmega328 nodes, wireless switches and nodejs on a linux system for control, DB storage and web interface. But this MQTT sounds good, and it would be great if I could replace some part of the NRFs by ESP8266s…now trying to understand the eclipse gui and how to create/adapt software for ESP8266

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s