A fine day for playing with solar panels


Please note – this blog is WELL out of date and all of my blog items and much, much more have now been moved to http://tech.scargill.net

7am here in Bedrock as we start our second full day in Andalucía.

It’s starting to look good already and this morning Maureen is off with some of the ladies to have fun while I sort out some tech work I have to complete via Skype with my friend Aidan in the UK.

testing solar panelsMy jobs for this weekend include getting the watering system back up and tested this time using an ESP8266 board with the Raspberry Pi controller which I’ve just spent several hours getting working thanks to a router that amazingly by default isolated users from each other (believe me that took some finding).. I’ve brought much more meaty power supplies (12v 6amp) with me to Spain this time to handle the rather unpredictable power here – though I suspect I’ve not brought enough.  As an experiment I brought another solar regulator along with an alarm 12v battery and panel –I want to see if I can run the router and  Raspberry Pi controller off the sun entirely. I suspect that’ll have to wait for our June/July run in which we’re bringing the car from the UK as I just could not fit my largest (40w) solar panel into the travel case!!

I’m quite excited today as the tech Facebook page just tipped 1000 likes…  but first, we need to get into that lake and enjoy the fine weather Andalucía is having right now, t was a little misty at 7am but already as 9am approaches, the sky is clear blue and will typically sail past 30c in the afternoon with peaks around 40c which might sound a bit much to Brits starved of sunshine but trust me – you get used to it if you’re neck-deep in a lake.

As you see below, my Arduino/NRF24L01 network has been working reliably over here, tracking temperature and humidity as well as giving me remote access to lighting etc. I have been logging temperature and humidity for a some time now) and the humidity is finally dropping rapidly – we picked a good time to arrive.

Graph of temperature and humidity

Just put the latest “Esp8266 unofficial” development kit on my laptop,  developed a MYSQL logging protocol for the little ESP boards and just about ready to start populating the place with controllers. I just need a little clearer understanding of the MQSQL  “Insert into… on Duplicate” code I’m playing around with here http://sqlfiddle.com/#!9/04c6c/1

For now I’m off with my OWNCLOUD-backed-up mobile phone, thanks to THREE network I have access to my unlimited data while abroad so no problem syncing up with servers. Nice.

Lua and the DS18B20/DS18B20P temperature sensor

DS18B20 or DS18B20PA long time ago, back in the dark ages when all we had were Arduinos (seems such a long time ago) I spent a bit of time playing with the Dallas DS18B20 chips.

There are a number of ways to read these and so I’ll explain my logic as I go along. Not everyone will agree with my conclusions and that’s fine. Works for me as they say.

The example that comes with the Lua interpreter caters for almost everyone by making the software check for missing chips, multiple chips and offering a range of output options including degrees C and F.  As the language subset does not include floats, it simulates that with two catenated numbers.

In order to avoid issues with delays stopping communications ( and I can confirm that delays WILL stop for example incoming MQTT messages), it also does away with the typical setup delay.

So –  a number of issues with this.  Firstly, for me, if I am reading temperature and maybe controlling something, I cannot for the life of me see the need for sub-degree precision. Accuracy yes but precision? Really, you’re going to keep the house at exactly 22.15c?   Probably not. I control 2 properties and this code will likely control the third – 24-7, 365 days as year if it works as reliably as the Arduino version.

Secondly, when I received my Dallas chips I got a mixed bunch on special offer and ended up with some with a P suffix. I quickly discovered that the delay is needed for these chips and today when using the example software with Lua I was set back temporarily before remembering this fact. The P suffix chips simply would not work AT ALL and constantly returned 85c. That delay is needed one way or another.

Because it has been removed, the first reading you take with the code will be incorrect.. and that’s not really a problem – you could take that during setup. There are ways around this but it’s not worth the bother when it’s so easy to circumvent.

Now personally I don’t see me using more than one chip per board and if I did I could always use another pin – so that whole search scenario is a waste of time, resources and RAM for me.

So given these criteria I have a MUCH simpler version of the driver for you – and one which will work without alteration with both chips. It’s also fast.

Instead of seeking the address of the chip, then starting the conversion, then reading the chip…  I assume one chip, simply read the value then start the conversion.

This seems utterly wrong but then remember your first reading is naff anyway. Assuming you don’t try to read the unit more than, say, once a second, the time in between reads will be more an enough to do the conversion for BOTH types of chip. I plan to stick this on a timer, say, every 10 seconds, dumping the value into a global variable so it’s “just there”.

Also as I’m just reading an integer, about the only check I have to do is for overflow so as to perhaps create a negative number (for freezing conditions) and finally all that CRC stuff and the buffer needed to store the data – why bother, with only one chip it’s not going to go wrong unless you have a long lead between the board and the chip. You only need to read 2 bytes!! Speed, storage…

So here’s the EASY version.

I have to say, I simply do not understand the registration system for LUA modules yet (wasn’t there last time I looked in) so those first few lines at the start I’ve copied verbatim.

Can you make it even simpler and also explain this stuff at the very first few lines to do with modname? Oh, I’m using all 3 leads (GND, D and VDD) of the Dallas chip and a 4k7 pull-up from D to VDD. I found the pull-up un-necessary on short leads but let’s not test fate?

Please note – this information has now been superseded by the info on the new blog at http://tech.scargill.net – please visit the new blog where you can search many hundreds of IOT-related entries.

— DS18B20 one wire module for NODEMCU
— LICENCE: http://opensource.org/licenses/MIT
— Vowstar <vowstar@nodemcu.com>
— Dramatic simplification: Peter Scargill

— Set module name as parameter of require
local modname = …
local M = {}
_G[modname] = M
— Local used modules
— Table module
local table = table
— String module
local string = string
— One wire module
local ow = ow
— Timer module
local tmr = tmr
— Limited to local environment
— Implementation – you don’t get any shorter than this

function readNumber(pin)
ow.write(pin, 0xCC, 1)
ow.write(pin, 0xBE, 1)
data = nil
data = “”
for i = 1, 2 do
data = data .. string.char(ow.read(pin))
t = (data:byte(1) + data:byte(2) * 256) / 16
if (t>100) then
ow.write(pin, 0x44,1)
return t

— Return module table
return M

That’s it!!! So there’s the library done… now here’s the test code…

t = nil
ds18b20 = nil

Notice that this, too is simplified. “4” refers to GPIO2 in the Lua tables. I’m using an ESP-01 and I use GPIO0 to control a relay.

and here are the results showing a swing across through zero thanks to my handy liquid air spray. I do wish I could figure out how to turn the UART off except when I’m firing prints out – anyone know how to do this? Results buried in that mess are –2, 0, 1 and are degrees C (you could easily mod the code to F).  Suggest not trying to measure over 84C.

> t=require(“ds18b20”)
> t.setup(4)
> print(t.readNumber())
> t = nil
> ds18b20 = nil
> package.loaded[“ds18b20”]=nil
> t=require(“ds18b20”)
> t.setup(4)
> print(t.readNumber())
0t = nil
> ds18b20 = nil
> package.loaded[“ds18b20”]=nil
> t=require(“ds18b20”)
> t.setup(4)
> print(t.readNumber())
1t = nil
> ds18b20 = nil
> package.loaded[“ds18b20”]=nil

If you’re on Facebook why not LIKE here. https://www.facebook.com/esp8266wifi

ESP8266 Temporary RGB Solution


Not exactly ideal (especially if one has already bought a load of WS2812b strip) but some time ago we had some little boards made to house 10 of the WS2812B chips – and while we were on we put a surface mount 328 chip on the back along with crystal and serial connector for an FTDI.  With the current issues getting accurate high speed timing out of the ESP8266 I bodged some code in the display board to look for {rgbdevice;red=x;blue=y;green=z} type messages.

I’ve modified the MQTT code already to stick {} around any message shown on the serial so it’s quite easy to pick out. The “rgbdevice” starter just gives it that much less chance of flicking lights for the wrong reason.

I implemented red, green, blue, bright, cycle, power, LEDs

The last one redefines the number of LEDs in we wanted to put several of these boards in series (obviously the others would not have the 328 mounted).  It’ll do until someone comes up with an assembly block for the displays in which case it should be possible to connect GPIO0 directly to the first display input.

The software for Arduino incidentally is simply the FASTLED library  – I stuck an infra-red sensor on for remote control but again the timing of the LED displays is such that if you’re doing a soft sequence as you see in the photo, the IR sensor interrupts get in the way. So maybe I’ll just stick to controlling the lot from my phone. Next challenge – get that OpenHab colour wheel working!!

ESP8266 and RB Serial LEDS

rgb ledsWell, it looks like everyone and their auntie have figured out how to get Arduinos to listen to ESP8266s sending out messages and control RGB serial LED strips – not surprising as there are several Arduino (and for other boards) serial libraries out there. I’m not talking about old fashioned 12v RGB strip but the more modern 5v variety with serial control, which, when (if) they get the price down, HAS to be the future of lighting.

I was at a hotel in Blackpool a few weeks ago and they’d just modernised and had the serial LED strip EVERYWHERE. If you’re not familiar with this stuff, each LED has a serial controller and can accept RGB settings – and once set can simply pass the info to the next LED. All of this is done with blindingly fast serial, so fast they build huge TV displays at airports using the stuff. Why is this stuff so much better than the old PWM red then green then blue LEDS? Because all 3 chips are in one housing you can create REALLY subtle lighting – you want daylight? No problem Tungsten? No problem? Nice warm candle-light? No problem.

Grab a strip, fasten it to an Arduino, provide plenty of 5v and a library and you’re up and running. You tell the library how many LEDS you have and it builds an array of 3 bytes for each LED. You populate the RGB info for the LEDS, call a function and BANG – they all get updated in an instant. with that array at hand it is trivial to write code to drive a colour wheel through it and by updating at regular intervals, create a moving colour change along the strip that is nothing short of stunning.

Right – that’s the easy way and also the larger and more expensive way. Now when is some clever person going to provide us with the same library (in C, not Lua please) for the ESP8266. Should not be a problem as the chip is way faster than an Arduino. I am not familiar enough with the chip to do this yet but I’m hoping someone else is – or has already done this – any takers?

Add this to the current MQTT code and you have an instant WIFI lighting controller. If antone has this or can make this I’m quite happy to put an article together – once I have my hands on a library (preferably working with GPIO0 or GPIO2 on the ESP-01 as that’s the cheapest and smallest solution out there but is pin-challenged).. it’ll take me no time to get that under message control.


This is fun – it’s a diversion but it’s fun – no… REALLY. Last night I was having a play with OpenHab.  I have all my own home control code and several working installations – but it never hurts to have a look and see what others are up to – especially having now pretty much decided to go down the MQTT route thanks purely to the fact that it works so well – and with passwords, in the ESP8266 hence solving a long standing security worry for me. I’m usually fairly good at spotting trends and as long as sending and retrieving MQTT OpenHabmessages in the ESP8266 nodes can be made rock-solid reliable – I cannot see any reason NOT to use the protocol – no worries about networks etc, just have the board respond to incoming topics and send out to reply topics as and when needed.   I can see someone coming up with a Windows interface for this (setting upt the router, setting up incoming topics etc  – and becoming a really popular person.

Anyway, I’ve been trying to get to grips with Openhab – trust me – if you’re new to it – you are going to need LOTS and LOTS of coffee.

The upshot is – it is possible despite the many many interfaces that OpenHab supports, to put 3 lines of code in there and have it send all it’s stuff to MQTTand respond accordingly.  So looking at the screen above, I can turn on the bathroom mirror (that’s the demo almost unaltered – I didn’t write this) and that will send an MQTT message out.  I have my little board responding to that. I can even have the board send a response BACK to say what the status of the mirror is (ON or OFF)  (I have no idea how to make that response happen on request – i.e. when the web page first opens).

ANYWAY, my little ESP8266 board can make the yellow bulb turn on and off but the blue switch sadly toggles regardless of any feedback – making you feel you’ve done something when in fact your little board could be in flames….. hopefully someone will help me there PLEASE. I turned off the yellow bulb automatic update but I can’t stop the blue button toggling yet.

SO – I got up early this morning and was waiting for someone to help me with my little Openhab issue when a fellow wrote in to suggest I have a look at CODE-RED. Well, what a fiasco – there’s a lot of LINUX-ASSUMPTION in the setup – just install it – no problem they say!  Well also install NODE which to be fair is also easy… but when the installation cran.. I ended up with a TON of guff coming up and telling me I needed to install VISUAL STUDIO…. I was THAT far off scrapping the idea but I thought I’d give it a go and despite the warnings it seem to work.

So what’s NODE-RED – I have NO idea if it is of any use but I CAN tell you it’s a HELL of a lot of fun – I mean – you HAVE to try this.

node-redSo if you’re bored and want some fun…  go here and get node.js -  installation on Windows is a breeze.  NOW having done that,  go off and get the NODE-RED installation.. http://nodered.org/  -  I just dumped the directory on my desktop.  Open a DOS BOX (they don’t tell you that) and navigate to the new node-redxxxx directory in your dos box.  Now type npm install –production

Ok now in the same DOS BOX – type node red.js and this still start up red.js – it will also likely complain that it has failed to register 9 node types – I have NO IDEA why a fresh installation would do this – and at that point I was ready to throw in the towel  -but it works anyway!!! If anyone has a kiddies guide to fixing this please do write in.

So – you will get to see something like this…

C:\Users\Peter\Desktop\node-red-0.9.1>node red.js

Welcome to Node-RED

9 Jan 07:05:17 – [red] Version: 0.9.1
9 Jan 07:05:17 – [red] Loading palette nodes
9 Jan 07:05:17 – ——————————————
9 Jan 07:05:17 – [red] Failed to register 9 node types
9 Jan 07:05:17 – [red] Run with -v for details
9 Jan 07:05:17 – ——————————————
9 Jan 07:05:17 – [red] Server now running at
9 Jan 07:05:17 – [red] Loading flows : flows_Monster.json
9 Jan 07:05:17 – [red] Starting flows

It might now look exactly like above….

I left the DOS box open – and pulled up my (Chrome) browser and ran the address and port provided.

A VERY pretty screen came up..


Actually if you don’t get this far, go back, to a little reading – I might’ve missed something – it really is worth getting this far..

You back? Right – see those buttons on the left – which kind of make you feel this could be tantalisingly easy? Well, it IS. AT this point I have to make some assumptions – you have access to an MQTT server somewhere – and you have a Google email address – and a Twitter account or some of that.

So here’s what I did – I dropped an MQTT input and a DEBUG output on screen – drag and drop. I drew a line connecting them.


I double-clicked the MQTT input (see it has the speaker-thing on the left as against the right) – and filled in the name of a subscription.. and my MQTT broker details.  I pressed DEPLOY.

Excited yet?


Using MQTT-SPY I sent a message to /test/ and …


By this point my toes were tingling… I wonder if I can have it send a message back as well?


Yup… but.. but… but… how about having tweets come in and go to the debug AND my /testreply/ topic? I eagerly dropped the TWITTER box in and authorised it (double-click – absolute doddle) – I put in some hashtags (without the tags it seems)…



and drew a couple of lines – hit deploy and…

twitter mqtt

Unbelievably, tweets started coming into the debug window – AND going out to my /testreply/ – no problem.

Let’s try an email I thought – could it send me an email as well – and what about the contents of a test button going to all 3!!


Sure enough… I pressed the test button – the debug showed the message – my MQTT /testreply monitor saw the same thing and an email came into my inbox.

That’s it – I’m going to leave the rest to you… the visual interface (browser) does not need to be running for this to work (well apart from the test button) – and I’m not sure what kind of kit you need to run to keep it going either – or how it would respond to power loss etc, but at least ON THE SURFACE OF IT this as something special! I messed around for DAYS  trying to get a crappy Arduino to send Tweets only to discover most of the code out there is for BEFORE Twitter changed the API – and here – I can have a £2 ESP-01 send tweets AND emails without any effort at all!!!

Back to my Openhab work for an hour then I have some real work to do.

Incidentally, I have a SYNOLOGY disk-station and thanks to helpful people in here I now have that running MOSQUITTO for the MQTT server and thanks to the incredible help from the fellow who put together MQTT-SPY and made loads of changes for me – I can keep a test rig open to test my MQTT. I’ve even sent 1000 messages a second to the broker and had them come back in within a second – so there is no doubt in my mind that this is the correct route … it’s looking more and more like there are a ton of useful toys like OpenHab and Node-red which will help make this a very satisfying project. It looks like I could put Openhab into the Diskstation as well (in case you don’t know the Synology Diskstation is one of these black boxes with 2 disks for storing your media – because they use RAID they are almost immune to failure (one disk fails the other takes over) and unlike the READYNAS before them – they have some power).  IDEAL tool for running stuff like this as they’re on all the time anyway.


ESP8266–a Way Forward

Right, it’s becoming pretty clear how this is going to develop for some of us… I’m not interested in joining someone else’s circus when it comes to home control – I want to do my own thing… but I want inexpensive mechanisms to help me do that and it’s becoming clear that MQTT as a communications mechanism is going to be part of that. It’s also clear that the ESP8266-type boards are also part of that.

So here’s where I think we are right now – but firstly I need to explain MQTT for absolute beginners before they run away:

I’ve been messing with home control over 2 years now, I wasted a year of my life on those AWFUL NRF24L01 boards which can’t even manage a single stone cottage wall and won’t send and receive at the same time. Then I moved onto better radios and even did some work with a friend on the Atmel radio processor boards – and then all of that seemed to change – the ESP8266 WIFI boards came about. Why were they better? Well first off they could use your existing WIFI infrastructure, they had excellent range, they were CHEAP – and by that let me make my position clear – I think spending £30 to turn a 50p lamp on and off is STUPID… it needs to be a LOT cheaper – like “a few quid”. When it comes to complexity – a little WIFI board on it’s own is sometimes all we should need (temperature sensor), or for a more complex item that feeding, say, an Arduino. In the past the problem has been that by the time you add Ethernet libraries and radio libraries to the Arduino you end up with almost no memory left.. and hence my use of the more powerful 1284 chips on custom boards – but that’s not REALLY the answer.. Micro-Arduino boards from China at £2 is REALLY the answer.  So a WIFI board+micro-Arduino for £4 or so – that’ll do.

The ESP-01 board is a perfectly usable board which sadly has a couple of things missing – the link for SLEEPING being one of them – the chip pins are way too small for most of us to solder and so it looks like the slightly more expensive ESP-12 might be an answer – it also has lots of FCC and other stickers to keep those people who care about such things happy.

But how do they all talk together? It seemed at first that the ESP8266 would do it all then reality set in – you can’t be a client, talking TCP/IP to a phone – AND be a web client, getting the time etc. at the SAME TIME – OH DEAR, the dream was starting to come apart. And then along came MQTT.  It’s REALLY simple… you have a machine or service somewhere called an MQTT broker and all it does is this.. everyone logs onto it with a unique name, lets say we call our devices peter and paul. Peter and paul can receive messages from the “broker” and send message – to anyone. So Peter can sent a direct message to paul. Problem solved. But it gets better.

The TOPIC system lets peter only subscribe to messages he’s interested in.  So, if Peter was an external temperature sensor he might only be interested in messages requesting the temperature.  If Paul however is a thermostat control, he might be interested in knowing what Peter’s temperature is – and what the time is . He might also want to know if he’s on PEAK time or not, or what the current requested temperature is. If this is not exciting I’m doing a piss poor job of explanation as it’s got me excited.  So whatever this protocol is it needs to be available on PCs and Linux etc., it also needs to be available on ESP8266 for those projects that use it – and on Arduino for those projects with a hardwired Ethernet connection. Well, as of around now – IT IS!

MQTT is now available on the ESP8266 but BEWARE – the author himself says this is for messing with and not yet for production – so don’t get TOO carried away – a new release came out yesterday with SSL. https://github.com/tuanpmt/esp_mqtt – my own experience is – it WORKS. I’ve had a little light sitting there dangling from a shelf, listening to the broker and responding – and it just keeps on ticking – loss of power no problem – it picks up where it left of.

But what if the little device is not on all the time – won’t it miss messages? Well no because there are 3 types of message (it’s important this is implemented in whatever you use. Type 0 message means send it and take pot luck if anyone gets it-  type 1 means they are guaranteed to get the message – but they might get it more than once. Type 2 means they will get the message for sure – once only.  This is nothing more for you to worry about than putting a number at the end of a function – 0,1 or 2.

So, let’s backtrack – that time thing. Right now I have an Arduino which calls a web page I wrote to get hte time because I don’t find NTP time servers that reliable (they get a lot of use) and they don’t send me lighting up times etc. I wrote my own PHP page sitting on a public server so I could get what I want, when I want.  And that was fine, it’s in the blog and some of you have used it… but there’s a BETTER way. Given an MQTT server and the common ability of web servers using CPANEL to allow “cron jobs” – WAY more complicated sounding than it is… imagine if I could have the ISP call that page every 15 minutes. Imagine now that the page instead of returning the time and light up times, sends that info to the BROKER!!! (As it will from sometime this afternoon until eternity). NOW the lonely thermostat ESP8266 in the kitchen doesn’t need a whole new protocol to learn – it just subscribes to that message and every 15 minutes it gets the time – SIMPLES!! But by now your ears are probably pricking.. hang on.. if I can do that then… yes, the humble web page now not only can send off the time, but a load of stuff like “lamp 4 in the living room – it’s time to turn on”. Instead of having to write a whole load of shite in a little processor you can have a password protected page on a website with TONS of terms and conditions to setup and you can update them on any old tablet and… well, I’m sure you’re writing code already.

What’s wrong with the above..

The little ESP8266 powers up for the first time, has no idea what your router is called.  So what we need is the MQTT software on the board, COMPLETE with the simplest page…. what is the router called – what is the password and what’s the address, name and password of the MQTT broker. As the ESP8266 can be it’s own access point this could all be done by a mobile phone which talks to the unit which ONLY is an access point if you pull a lead down to ground – or better – only when it actually has no information stored or the info stored does not work. Anyone up for that.

Lua allows people with no engineering degree to interact with the ESP8266 boards – and it’s very nice and there’s one fellow suggesting a way to merge MQTT and LUA – that would be great – but again BEWARE – the Lua installation WILL let you down if you get too ambitious – the implementation is unforgiving when you make mistakes and it EATS RAM.

If we could combine ALL of the above – a Lua interpreter that has it’s own web setup page – and the MQTT client software and maybe a range of temperature sensor libraries (the DHT22 is good – reliable, accurate and gives both temperature and humidity)….

BUT we won’t always be working with the ESP8266 alone – we might want it to be part of a package with an Arduino.. SO –  the unit when it receives messages should always put them out onto the serial line – maybe wrapped in some unique combination of characters – like << and >>or similar so it’s a doddle for the Arduino to pull out the message from any other crap (like the ESPs start-up 78kbaud crap we can’t seem to stop) with the absolute minimum or parsing… so for example if we have a topic peter/stuff and a message “turn something on” – maybe <<peter/stuff:turn something on>>   – see what I mean – look for “<<” – store until the colon in one buffer then till >> in another – and you have 2 buffers – topic and message – SIMPLES!!  But then there’s the slightly more complicated bit – they Arduino would need to send into the serial port two things…. 1. a topic/message in the same format – but ALSO commands to subscribe or unsubscribe to topics…   I honestly think at this point that this is ALL we need from the ESP8266… that of course and the whole thing being in a Windows- Eclipse compatible GITHUB package we can drop in – open up ONE web page in which we put our own special bits – using the Espressif SDK manual which they seem to be rather reticent to part with.

MOSQUITTO is a free MQTT broker (or server if you like) and it’s fine – but it’s all command line and pretty gruesome for beginners to understand. It can run on a PC under Windows… it’s ACTUALLY not that difficult once you’ve used it but messing with that config file is like a trip back into the 20th century – someone needs to make a nice sparkly Windows screen with lots of tick boxes.  When it comes to clients on the PC I’m in love with the free MQTT-SPY – it’s just lovely.  As for Android – really – MyMQTT is awful – if anyone wants to re-write MQTT Spy for Android they would be on my Christmas card list for years.

So there you are – I’m playing with Mosquitto and Rabbit-MQ which looks really nice but I have to say it did collapse when I tried to send a level 2 message out… might just be me not understanding things yet.

From here I’m ready to start coding – there’s enough above to form the basis of a very large and complicated control system –  I’m THAT far off having an internet controlled SAD light for the wife… it’s sitting here waiting for commands but I need something on the phone with nice round, shiny buttons and flashing lights that looks the part – not sure Basic 4 Android is going to do that for me but we’ll give it a whirl.

Let me know if this didn’t make sense – because if it didn’t you’re missing out on lots of potential fun.



Peter Scargill

p.s. The easiest way to get to grips with this MQTT thing – get MOSQUITTO – install it  – ignore usernames and passwords for now.  Get MQTT Spy – tell it about the IP address of the machine with MOSQUITTO (can be same machine) and PLAY… subscribe to arbitrarily named topics – send messages to those topics – within minutes it will all become clear. 

p.p.s. If you’re a Facebook fan and want to read about this stuff ONLY for Windows users who like pretty interfaces like me – head over to www.facebook.com/esp8266wifi

ESP8266 and MQTT A Marriage Made in Heaven?

It’s been a very busy day – and a very frustrating day. I’ve had to make a start at figuring out how MAKEFILES work because of a fellow who implemented MQTT for ESP8266 so I could seamlessly compile his code and blow ESP-01 chips with the Eclipse environment in Windows.

It all started a couple of days ago when I utterly accidentally stumbled on a project to implement MQTT onto an ESP8266. I don’t know why I went for this but I’d just gotten to grips with the Eclipse setup so I was now a fully fledged Windows developer of ESP8266 software.. by that I mean I’d figured out how to alter code to get a light to flash on and off in the AT command software.

I remember playing with MQTT a while ago and I mentioned it in here or on the ESP8266WIFI Facebook page. Anyway I sat down with a large glass of wine and started to tinker. Essentially, MQTT is a lightweight protocol for sending messages around a network – apparently Facebook Messenger uses it.  You have a “broker” – call it a server if you are a techie… so this is free software like for example MOSQUITTO which you can run hosted, on your PC or on a Raspberry Pi. you can make an account with it – which might be open, have a simple password or have something far more complicated.

One of my fears of controlling the house remotely has always been security and I’ve made some primitive passwording which is really not worth a light.. so this caught my interest. My other issue as regular readers will know was that I was getting utterly hacked off with the ESP8266 as it can’t really do server and client at exactly the same time – so you would have trouble making a network of them – you CAN do it and there’s a slow moving project on the subject… but right now, not a viable solution – that might change.  The instant I saw MQTT my heart skipped a beat as I thought… can’t one device send a message to another?

I put IBM’s free MOSQUITTO on my PC (powers up with no passwords etc, doddle to use even if the command line environment makes some people feel physically sick – including me). IT uses a single port which you can change so it’s easy to make available outside via a little port mapping on your router.

I blew the code for this onto one of my ESP8266 boards – and left that connected to a serial monitor. Sure enough, it connected first time to MOSQUITTO. It SUBSCRIBED to a couple of messages (utterly arbitrary and they don’t need to exist at the time of subscription).

I grabbed a free but not very good program for Android called MyMQTT. IT’s not seen an update in AGES but I remembered it worked ok. I told it about my MOSQUITTO setup (ie the IP address of my computer) and that was that. I subscribed on the mobile APP to the same “topics” my ESP8266 had. I sent a message to that topic… sure enough instantly on the phone, the topic I’d replied to came back with a message – the same one I’d sent out… simultaneously the same message came into the ESP8266. We’re talking INSTANT here – no visible delays.

Immediately I grasped the possibilities. I modified the MQTT code (in the new sparkly ECLIPSE environment) on the ESP8266 to check the content of one of the topics it had subscribed to… a quick STRSTR on the zero-terminated incoming string… I made 2 comparisons – one for a message “GPIO0 ON” and another for “GPIO0 OFF”. I borrowed the heater and port info from a BLINKY sketch (all of 3 lines of code for everything)… and compiled away. I sat in the corner of the room and gingerly entered “GPIO ON” into my mobile phone app… and… BINGO the light came on.

Step 2. When the light comes on – SEND a message to the same topic but a different device number – I should say the author engineered auto device numbers based I think on the mac codes – they’re all unique…   /0033342342/data as it happens – I sent a hello.

Sat back with the phone, put my GPIO message in again, the light came on and IMMEDIATELY the message appeared on the SECOND ESP8266 as expected.

So with the minor inconvenience of a piece of “broker” software minding it’s own business on a minimal PC, a Raspberry Pi or some external server (your choice) the ESP8266 boards can now talk to each other without another processor. Not only that, I reasoned but if the server machine also had a client, it should publish a “TIME” message every 10 minutes or so…  the boards would IF APPROPRIATE be programmed to subscribe to this channel and LO  – ESP8266 boards which can turn things on and off – and also know what time it is. But what of the power of an Arduino – what about all those extra pins? Another project implements I2c on the ESP8266 and when my skills are ready I’m adding THAT into a mix – which means any number of peripherals to do things or return info using MQTT.

It’s a grand total of £40 or so investment to get yourself into the Android App trivial programming brigade using B4A.  Mine of out of date, out of warranty and so I’ll ahve to start again. For £10 a guy has written a library for MQTT and this afternoon on a friend’s PC – someone who has the full B4A, we wrote a simple messenger program that will send and receive MQTT messages. The results – again absolutely instant message passing back and forth.

Imagine – you have a button on the phone – you press it and it does two things.. it turns a little indicator called kitchen_lamp to grey then sends a message to /kitchen/lamp which it does not subscribe to – the message is ON… the kitchen lamp ESP8266 box which features nothing more than an ESP-01, a 3v mains PSU and a relay, gets the message and turns off GPIO0 which turns on the relay (there’s a good reason to do it backwards like that). It then sends a message back to another topic which it does not subscribe to called /kitchenresponse/lamp. THe phone however DOES respond to that – and when it sees an incoming message to that topic, it turns that same indicator green or red depending on the message.  So the lamp is off – you press the on button and your indicator on the phone almost immediately turns green so you know for a FACT that the kitchen light came on.

You might take that a stage further – because you’ve written the client on your PC to send messages like TIME, the kitchen lamp knows what time it is because it also subscribes to TIME.  if no-one turns that lamp off, at midnight it turns it off itself.

Multiply this by dozens of similar gadgets, some DO stuff like turning things on and off, others merely sit there, read the temperature and report it back – either every now and then – or on demand… same technique..

An old Android tablet sits on the wall, it has a program you wrote which subscribes to as many or little of the topics as you want and reports in pretty graphical format the various statuses-  it might even have a large round touch control reminiscent of a a certain apple-esque temperature controller…. and everyone talks to each other securely over WIFI..

BUT you might say, I really want to use Arduinos – ok you have the little ESP8266 do all the hard work of handling messages incoming messages it sends out of the serial port and if you want to send a message that could come in through the serial port… so now you have an Arduino that has to do nothing but take in serial messages and send them out – polling is easy – and you can of course wrap messages in any control character or sequence you want.

But hang on – isn’t Arduino serial interrupt driven? YOU BET – which means if you go in and tinker with that you could have that look for the starting sequence, start storing bytes until the ending sequence occurs, copy that temporary buffer to another, set a flag to say there is a massage – and simultaneously go back to looking for the next message – all using existing resources that are used in the serial handling anyway.  The Arduino ends up with a few bytes less than a full pack and can now receive serial messages and act on them with virtually no extra overhead which means the simple 328 has LOTS of room to DO stuff.

That enough for now? It certainly has me excited but I’ve had enough for one day… I need to turn most of this from thought processes to reality. The MQTT project is apparently in it’s early stages but I can assure you it sends messages with password protection as reliably as it receives them. Even if the designer does nothing else you have this at your disposal – links in earlier entries in this blog. Go get ‘em.

Now if anyone’s up for some collaboration I’d much rather someone else write the Android APP – a decent replacement for the promising but dead in the water myMQTT..  All in good time – good night all.

ESP8266 Development in Windows AT LAST

ESP8266 on WindowsHere it is… on the surface of it a MASSIVE Christmas gift to those of us who simply have to little time on our hands to get involved with Linux.

I got so frustrated waiting for up to date BIN files and the continuation of Chinglish error messages (I should not complain because many Chinese speak far better English than I speak Chinese) – I decided this morning to down tools and see if I could get the compiler working in Windows – but I wasn’t expecting THIS.

I was stumbling through various blogs and discovered THIS.

http://www.esp8266.com/viewtopic.php?f=9&t=820&sid=783b2f3021078ab65722108fdfadd5d8 – the link gives you all the files you should need to get working environment in 64 bit Windows – and this video https://www.youtube.com/watch?feature=player_embedded&v=eWGjADdjH98  shows  a demo in the video of the whole ESP8266 compilation and rom-blowing – WITHOUT reams of irrelevant warning messages!!

ESP-01AT LAST, ESP8266 compiling and blowing in human-readable form. I got VERY exited about this as there are several developments out there which are ALMOST what I’d like but not quite – a stress free way to mod the source codes would be lovely. For example one fellow has developed the AT command set for ESP8266 using the latest SDK and added NTP support – another has added a well needed internal buffering scheme to the serial – what would be great would be BOTH of these in one package – the only way to do that is to get the sources, compare, mod and blow your own chips – and now my Windows friends – you CAN.

Check out the link and follow the instructions EXACTLY – don’t go off on a tangent as there’s a similar page with failed links and lots of Russian – you’ll find the JAVA file on the linked website is later than the one he specified – won’t worry about it – still works. There is an issue also that in his diagram he shows the ESP-01 with pins GPIO0 and GPIO5 connected to the FTDI (USB-RS232 convertor) – well, you don’t HAVE GPIIO-5 available on the ESP-01 – I’d tell him that but I don’t speak Russian – I did leave a note on his YouTube site. . Matters not – you merely need to ensure as before that GPIO-0 is grounded before you start blowing chips – and don’t forget to cycle the power to the board before and after.

Follow his instructions TO THE LETTER. I installed each of the packages (with the updated Java package – when it came to his EXTENSION package which consisted of a BATCH file, I noted (typical command line stuff – whizzes by you too fast to see) some errors to do with packages already installed. Mad no difference, still worked.

Eclipse on Windows compiling ESP8266I followed the instructions for the Eclipse Luna program including point 8 in his documentation though that last paragraph I simple could not understand. But then I realised this was explained in the short video.

So – 5 files in all totalling maybe 400Mb – and worth every byte!

By the time I was finished (NO extra work other than following the instructions and no “assumptions” other than having Windows 8 64 bit)… and I ended up with this.. As you can see, pretty damned impressive,  a working rig with various projects already included on the left, all set up INCLUDING the latest AT v0.2.0 on SDKv0.9.4 – which is the one I’m currently interested in. No Lua but I’m sure someone can help there.

Compilation WORKED (wheeeee)…

tmp26DDBut then when it came to the last bit – blowing the file in to the ESP-01 – FAILURE.

I found at least 2 places in the MAKEFILES referring to COM2 – which I changed to COM5 (all done within the environment) – that just seems the WRONG way to do this – surely you would set the COM port outside of the MAKE file and pass it as a parameter – if you figure this out – please WRITE IN… 

I changed the COM port manually and eagerly double-pressed the button to blow the chip. My little board flashed for a second and… failure. I’d forgotten to short GPIO-0 to ground. Power off then back on with the link, plug into the PC via my FTDI –and…..SUCCESS. 

Right about now I can make a minor mod then compile and blow all within 26 seconds – which is amazing.

ESP-12So – there you have it – that and the revelation that my FTDI can indeed generate it’s own 3v3 and hence I can run the Arduino and the ESP8266 on 3v3 instead of messing around with level convertors…. it’s been a GOOD DAY so far. No longer the poor cousins of the Linux crowd..

I hope you get as much fun out of this as I intend to. Next stop I need to ensure you can just pull in a GITHUB folder structure as I’m sure these won’t be ENTIRELY up to date… he doesn’t seem to have the LUA interpreter in here – as that has only JUST gone open source – again if you manage to pull that in and compile and blow, please come back in here and give us a step by step along with any other news of updates to install – I can see this environment getting a lot of use.

I installed this on 64 bit Windows 7 without ANY issues, next stop I hope to repeat this with Windows 8. If only Dropbox was faster…

07:04PM – Tested in Windows 8.1 64-bit – no problems – same procedure, same results – success.

08:02PM – Tried the MQTT demo – loaded it in from the site ZIP, imported all fine, compiled all fine but did NOT generate a firmware directory and so blowing the files could not proceed. Maybe this is a one off problem?

If you’re on Facebook don’t forget to LIKE my Facebook page – where I can concentrate purely on Windows users.. https://www.facebook.com/esp8266wifi

ESP8266 Mesh Network

mesh networkI don’t want to raise false hopes but there’s a fellow over at http://www.esp8266.com/ who claims he’s getting on with creating a mesh network with ESP8266 boards. If he can do that AND have one of them talk to the outside world – for example an Android APP using TCP/IP — then I’m throwing my radio network in the bin. This would have to be very reliable but given the rubbish range of most of the little radios and the fact that most places are wired for WIFI… there are just SO many possibilities for this. I’ll be keeping an eye on this chap  – look in after Christmas for more.

08:49AM – You’re going to love this – he has something to demonstrate – and the more I think about it – the more I think he’s onto a winner. The units as you probably know are able to be access points – and clients at the same time.  You may also know that you can get a list of available access points and signal strengths nearby. This is where the NRF24L01 devices go wrong – you can’t do that.

esp-01So given that you can scan for the best signal – and given you could have a given range of access point names which you  would know about – and a passwords you’d agree on, then is it really that far fetched to imagine a unit regularly checking access points and hooking up to one. Starting to make sense already? I’ve not thought this through, I’m hoping the fellow who’s writing this has.  I can’t get my head around how they’d actually talk to each other AND the outside world…. ok, they can be TCP/IP listeners and handle multiple connections but using serial I could never get the multiple connection bit to actually work. Maybe it’s better when you’re not using serial.

If nothing else I intend to set up 3 of them and follow this project through… lots of potential here I think. Some exceedingly preliminary code is available already to play with…

09:48AM – Not looking good up to now – followed the instructions here.. http://www.esp8266.com/viewtopic.php?f=6&t=903&p=5581#p5581  pretty much to the letter… checked my locations, blew 2 chips – and… nothing – not talking at 115,200 or 9600 on serial, not showing any thing new up on my network…