ESP8266 the unstoppable March of Progress

esp01I’ve been following the progress of this little chipset as you know since day one – having spend a year more developing skills at using horrible little NRF24L01 radios with as much indoor range as my legs – and Arduino Ethernet cards (which to be fair are WONDERFUL other than their need for a marriage-endangering WIRE), out of the blue came a little Chinese board with the potential to change everything. The ESP-01 showed much promise but very little reality at first. The instructions were in Chinese, the documentation was in Chinese or missing, people told me “there are these other boards for only a tenner that do so much more”.

And yet here we are today with much of that about to change. The new so called white ESP8266 board with it’s development “kit” for less than a tenner is winging it’s way to me now – and the ESP-12 boards are neat and even have approval stickers to keep people happy not to mention a health range of I/O.

There are now a range of boards, all cheap  (there is a question mark over the ESP-07 – not for the first time I’ve heard there are 2 versions one of which is “suspect”) – and the ESP-01 remaining, I think true to my original statement – the title of one of my blogs “the cheapest computer in the world”. At under £2 tell me I’m wrong. 1 plug in the wall power supply , one ESP-01, one cheap solid state relay and you have a functioning WIFI-controlled device – with the right software that is.

But like all such devices, software makes or breaks this dream and in recent weeks we’ve seen the emergence of the ECLIPSE environment for Windows – I’m sure there are many such environments for Linux – and others will blog about them – I’m not interested in the command line and haven’t been for a long time. Eclipse and the tremendous work by the fellow who set this all up for us – means everyone from expert to complete beginner can “have a go”.

This week however is a bit special – we have in one corner TUAN and his MQTT code – on the other corner we have ZERODAY and his LUA interpreter – ( I really should find out everyone’s proper names) and this weekend – they look like merging.  I’m hoping that the MQTT code will continue to be developed independently of LUA as I think there are uses for both. I DO have code that WORKS – it’s not just a promise. We have some great tools such as Esplorer for testing Lua and the AT command set.

I can’t tell you what is possible with the software and hardware out there – all I can tell you is what I’m doing and why – I hope my blogs up to date have given you all the links you need. For days now I’ve been testing the MQTT software – and as far as I can tell there is only one “bug” left of note and it’s not even a bug – it’s an inconvenience – something to “wrapped up” – essentially the code works – but it’ not too happy about empty messages. I expect right now that is being fixed and I am so grateful to so many authors who have communicated, helped, changed things – in the main for no commercial gain – because they love it.. it’s almost like the days of the first 8-bit micros – the rush of learning.  Meanwhile I got an email this morning to say that MQTT and LUA were merging and a test set of .BIN files was available.

So, despite a head that is spinning like a TOP due to flu, this afternoon I set up a test rig…  let me take you through that quickly… erm… it works but read on as you need the bits that come before it.

The plot so far:

In order to use MQTT as the base of your control system (in my humble version of this vastly diverse Internet of Things) you need an “MQTT broker” running somewhere and it needs to be reliable – ROCK SOLID. In my case I have a Synology Diskstation (the link is an example – mine is not white) and with a little help I got the free MOSQUITTO running on it – you could put this on an old PC, a Linux box – just about anything with a brain – but it needs to be reliable. There are services out there – the free ones they generally warn you not to rely on them – so I’m not going to – I’ll have my own thanks – I’ve talked about several variations elsewhere – at the end of it I went with Mosquitto.  My disk backup DISKSTATION is on 24-7 so it seemed the ideal place for this. As for testing Mosquitto on a PC – you are not going to beat the EXCELLENT MQTT-SPY.

So now a place to fire messages to – and a place to receive them from… and the software tools for ESP8266 to send and receive those messages – the sky is the limit. But what about issues people have had – swapping modes – you might need the time from a time server for your little board or some other info. Well, I found a way around that.   MOSQUITTO is on all the time – what is needed is for it to make available info to the little boards, be it time, lighting up time, perhaps other information.  I found a PHP library – the nice kind – the type you don’t actually have to understand to use.  Some time ago I documented a web page I was using to fire back time and other information – I used it as I found NTP time servers to be slow at times.  I have cheap web space with a provider much as many of you will – with CPANEL – which means I get all the benefits of using a reliable Linux based web service while staying within my pretty coloured boxes visual interface comfort zone (I’m sure there’s an abbreviation for that).

With such a setup you can write PHP pages like one that will send time messages for example and with the ability to run those pages on demand using CRON (doddle) you can make the time and over information available to your toys.

Here’s the deal.

<?php
require("../phpMQTT.php");

$mqtt = new phpMQTT("xxx.xxxx.org", 1884, "somenamePub");
if ($mqtt->connect(TRUE,NULL,"mymqttuser","mymqttpass")) {
            $locn="Europe/London";
            $lon=55;
            $lat=-2;
            date_default_timezone_set($locn);
            if ($_GET[‘loc’]!="") $locn= $_GET[‘loc’];
            if ($_GET[‘lat’]!="") $lat= $_GET[‘lat’];
            if ($_GET[‘lon’]!="") $lon= $_GET[‘lon’];
            $dateTimeZoneLocal = new DateTimeZone($locn);
            $dateTimeLocal = new DateTime("now", $dateTimeZoneLocal);
            $localDateTime = date("H:i l d-m-y", time());
            $localTime=strtotime($localDateTime);
           
            $mqtt->publish("time",$localTime,0);
            $mqtt->publish("timestring",$localDateTime,0);
            $sun_info = date_sun_info($localTime, $lon, $lat);
            foreach ($sun_info as $key => $val) {
                if ($key==’civil_twilight_end’) $mqtt->publish("dusk",$val %86400),0);
                if ($key==’civil_twilight_begin’) $mqtt->publish("dawn",$val %86400,0);
            }
    $mqtt->close();
}
?>

No knocking my coding please – I do not profess to be a PHP expert – indeed – can you think of decent extra publications this page could produce for the little gadgets? If so lets have those ideas! 

This web page connects to your MOSQUITTO or other MQTT broker and every minute (every week if you want – my choice of every minute – gadget turns on – worse case it waits a minute to know the time) I am here publishing 4 items – the TIME in standard seconds-since-1970 format, British lighting up time (DUSK) in seconds since midnight, DAWN in seconds since midnight and TIMESTRING for those with no space in their project to format the time. ANY of my gadgets can choose to subscribe to these  publications – or not. And as you can imagine I’m planning more publications – timed an otherwise. The server load of the above is irrelevant – I don’t see anyone’s provider griping about this.

So – the people I’ve been with keep my web pages up and running 24-7 and and have done for as long as I can remember – why would I not trust this info.

And all of this brings us around to LUA+MQTT – possible only since this morning!!!

Check this out. Sorry it’s not formatted.

mqtt = net.createConnection(net.TCP, 0)
— init mqtt client with keepalive timer 30sec
mqtt:mqtt("myid", 30, "mymqttuser", "Tmqttpassmakeitagoodone")

— on publish message receive event
mqtt:on("receive", function(conn, topic, data) print(topic .. ":" .. data) end)
— on connection event (all event inherit from net module)
mqtt:on("connection", function(con) print("connected") end)
mqtt:connect(1884,"mymqttserver.address")
— subscribe topic with qos = 0
mqtt:subscribe("time",0, function(conn) print("subscribe success") end)
— publish a message
— mqtt:send("/topic","hello",0,0, function(conn) print("sent") end)

print (node.heap())

mqtt:subscribe("temperature",0, function(conn) print("subscribe success") end)

mqtt:send("dusk","hmm",0,0, function(conn) print("sent") end)

THAT which you see above and the code to log into your access point (which, once entered should stay there) is it – that code is all I had to put in – to have the little board be subscribing to time, dusk, dawn and temperature from another device (oh – a pair of minuses means comment – you need to see how to publish).

The REST is what you do with such info – Lua has timers (to keep that time going if you lose the internet connection) and GPIO control – to do things – already discussed previously – I’ve had a solid state relay running directly off the board and powering a mains lamp.

And that is it for now – the information isn’t here – it’s out there – check out the links here and in my previous blogs and bear in mind that before Christmas I knew NOTHING about compiling this stuff and I was determined determined not to get into Linux (and still successfully avoiding it)  – I’m now just about ready to openhabproperly embark on some real projects with real control in the comfort of my pretty-coloured Windows environment…  I hope that by putting all of this in one place gives you a head start.

For my next challenge – OpenHab but I think I’ll wait until the flu has gone – this one’s not going to be easy.

Advertisements

28 thoughts on “ESP8266 the unstoppable March of Progress

    • Appreciate that Jimmy – but up until Christmas for those of us not immersed in Linux, what seemed to be on offer were various text-based Linux emulation setups which if you’re used to the Windows environment and trying to learn something new (the new processor, new instructions, new API etc) was not at all easy. Now we have something that is at least familiar. Mind you a “Dummies guide to the ESP8266 API” would help as well but you can’t have everything. For my part I at least now feel able to make a start in using this knowledge to actually DO something useful with the chips.

  1. I checked out the NodeMCU GitHub page and didn’t see any updates that indicates that the MQTT Lua module is now available (although I knew they were working on it). I must be missing something, but where’s the updated NodeMCU code? Is it still in alpha stage and not released yet?

  2. Shouldn’t date_default_timezone_set($locn); be after the if ($_GET… lines?

    And you can use:

    function varcheck($var, $default)
    {
    return isset($var) ? stripslashes($var) : $default;
    }

    $loc = varcheck($_GET[‘loc’], “Europe/London”);
    $lon = varcheck($_GET[‘lon’], 55);
    $lat = varcheck($_GET[‘lat’], -2);

    for a bit more safety. 😉

    • There is a mis-closed paranthesis in dusk publishing line: if ($key==’civil_twilight_end’) $mqtt->publish(“dusk”,$val %86400),0); so I had to remove ) after 86400, other than that it works perfectly.

      I didn’t know anything about MQTT, but thanks to you I can now use it to send small messages between my homemade automation applianes. Thank you very much for your effort. 🙂

      • You know, I have no idea how that extra ) got in there – mine is returning the correct results- but you are of course absolutely correct. And since I put that page up it has now sent those messages 1384 times – without a hitch. I just need to get to grips with storing info in FLASH (I was saving that challenge up) in the ESP8266 so that the unit will continue to function if WIFI data is lost for a while.

  3. My thanks to Ozay – I would have found out soon enough about the time zones when I got to Spain and found that version lacking – and thanks for the code – my PHP is not stunning as you can tell. SO – the updated and fixed version of all of that is this..

    connect(TRUE,NULL,”myMQTTloginName”,”myMQTTLoginPass”)) {
    $locn = varcheck($_GET[‘loc’], “Europe/London”);
    $lon = varcheck($_GET[‘lon’], 55);
    $lat = varcheck($_GET[‘lat’], -2);

    date_default_timezone_set($locn);
    $dateTimeZoneLocal = new DateTimeZone($locn);
    $dateTimeLocal = new DateTime(“now”, $dateTimeZoneLocal);
    $localDateTime = date(“H:i l d-m-y”, time());
    $localTime=strtotime($localDateTime);

    $mqtt->publish(“time”,$localTime,0);
    $mqtt->publish(“timestring”,$localDateTime,0);
    $sun_info = date_sun_info($localTime, $lon, $lat);
    foreach ($sun_info as $key => $val) {
    if ($key==’civil_twilight_end’) $mqtt->publish(“dusk”,$val %86400,0);
    if ($key==’civil_twilight_begin’) $mqtt->publish(“dawn”,$val %86400,0);
    }
    $mqtt->close();
    }
    ?>

    Lets have some thoughts from others – what really useful information could be published this way for our little boards to save them some resources.

  4. Since you are using MQTT, connecting your devices to OpenHAB is fairly simple for both getting and displaying sensor data as sending data/commands to your devices.

    Have fun with it!

    • oh I expect I’ll get to grips with MQTT – it would be nice if there was a video or series of videos –

      Number 1. Adding a simple relay device via MQTT
      Number 2. Obtaining feedback from a simple device via MQTT
      Number 3. Adding rules one at a time..

      Actually having someone on the end of SKYPE with a screenshare would speed up development by 100 fold… still – can but dream.

      • Simple MQTT items, sitemap and rules are no problem for the three points you mention.

        And I’m running OpenHAB on Windows 8.1 btw…

      • Simple MQTT examples for items, rules and a sitemap is no problem fir the three points you mention.

        And I’m running OpenHAB on Windows 8.1 btw…

    • Ohhh Noooo. don’t tell me I’ve re-invented the wheel AGAIN… Anyone able to come up with the content for a rules and sitemap for a simple lamp with this feature – so I don’t have to read reams of documents – I’ll return the favour by blogging the experience.

  5. Another great post. I’m jealous – where are you finding the time to play, let alone blog at length about it?

    This week the few hours I had to play were marred by the fact that I didn’t clean out my esp_mqtt project directory before bringing in the latest version; as a consequence the ESP8266 would initialize then scream “malloc assert!” and stop dead. Just solved this this morning. Hopefully this will help anyone else with the same issue.

    MQTT, ESP8266 and esp_mqtt rock! A great platform to build upon.

  6. Very late unsociable hours is the answer to your question Ken. Blogging takes me no time, thats’ the easy bit. Hang fire as I’ve spotted a bug and reported it – it looks like you can only subscribe to 2 items at once – now this was originally hardwired into the code and I’m wondering if there is some legacy in there. It seems to apply to both stand-alone and Lua-embedded version. I found this out by elimination. Anyway due to time differences we won’t hear anything until tomorrow and I’ll be packing. And as you ask about time that’s mine about up. Off to Europe then London and my next stop at the PC will be Friday night if I’m lucky! I am pondering taking the little board with me as I’ve lots of wasted time between flights etc – but hooking up into an access point might be an issue. Our little boards don’t handle web-based sign-on in hotels!!

    • I’ve forgotten the name of the program but there is a great one for turning your laptop into a repeater for hotel wifi. Since I’m pretty such it uses NAT on all your traffic the esp connecting to it won’t have to authenticate.

      • Connectify is the program I’m thinking of. It used to be free (and still has a free version) but it looks to be demoware now. Still probably useful enough.

  7. No there’s another one – Connectify was free and good – then it went commercial as you say and I think the demo is no good… I just need to remember the name of it.. I just feel it in my bones that the code is going to get fixed tomorrow and I’ll be stuck in London airport with nothing to do for 2 hours wishing I could try the code out. Once that’s done there’s a couple of mains power switches and a thermostat waiting for this code 🙂

  8. Peter,

    I made some examples for you. I can’t attach files, so I’m just copying them into this reply. If you want the files, let me know…
    I have no idea how the formatting will be…

    First of all. Say you have OpenHAB installed in C:\OpenHAB\runtime. In the subfolder configuration all the configuration files are stored:
    [installdir]\configurations\items
    [installdir]\configurations\persistence
    [installdir]\configurations\rules
    [installdir]\configurations\scripts
    [installdir]\configurations\sitemaps
    [installdir]\configurations\transform

    You do need the following bindings in the [installdir]\addons directory:
    – astro
    – mqtt
    – persistense.mysql

    The sitemap file scargill.sitemap in the sitemaps directory:

    sitemap scargill label=”example”
    {
    Text item=Date label=”Date [%1$tA, %1$td.%1$tm.%1$tY]” icon=”calendar”

    Group item=gAstro label=”Astro” {
    Text item=Astro_Sunrise_Time
    Text item=Astro_Sunset_Time
    Text item=Azimuth
    Text item=Elevation
    Chart item=gAziEle label=”Sun” service=”mysql” period=W
    }

    Text label=”MQTT Example” {
    Text item=SomeNumber
    Switch item=SomeSwitch
    Switch item=OtherSwitch
    }
    }

    Then the scargill.items file in the items directory

    //Astro Binding
    // ——————–

    // Astro items group
    Group gAstro
    // Astro azimuth and elevation group for persistence and chart example
    Group gAziEle

    // Sunrise and Sunset times.
    // Note that the offset= is the offset in minutes. This can be negative.
    DateTime Astro_Sunrise_Time “Sunrise [%1$tH:%1$tM]” (gAstro) {astro=”planet=sun, type=rise, property=start, offset=0″}
    DateTime Astro_Sunset_Time “Sunset [%1$tH:%1$tM]” (gAstro) {astro=”planet=sun, type=set, property=start, offset=0″}

    // Azimuth and Elevation are persisted and displayed in graph. To show both items in a chart, one must use a group!
    Number Azimuth “Azimuth [%.2f]” (gAstro, gAziEle) {astro=”planet=sun, type=position, property=azimuth”}
    Number Elevation “Elevation [%.2f]” (gAstro, gAziEle) {astro=”planet=sun, type=position, property=elevation”}

    Switch Sunrise_Event “Sunrise” (gAstro) {astro=”planet=sun, type=rise, property=start, offset=0″}
    Switch Sunrise_Event_Plus15min “Sunrise15m” (gAstro) {astro=”planet=sun, type=rise, property=start, offset=15″}
    Switch Sunset_Event “Sunset” (gAstro) {astro=”planet=sun, type=set, property=start, offset=0″}

    // MQTT Binding
    // ——————–
    Group gMQTT

    // Some number. Use ‘mosquitto_pub -t scargill/number/some -m 2.34’ for example to change the value to 2.34
    Number SomeNumber “Some Number [%.2f]” {mqtt=”<[homebroker:scargill/number/some:state:default]"}

    // If Some Switch is changed on the MMI. The ON or OFF command is send to mqtt topic
    Switch SomeSwitch "Some switch" {mqtt=”>[homebroker:scargill/switch/some:command:*:${command}]”}

    // The other switch. Can be set by changing the value of SomeSwitch on the MMI, but also using the following command:
    // Use ‘mosquitto_pub -t scargill/switch/some -m ON’ to switch the switch to ON, or OFF to switch to OFF
    Switch OtherSwitch “Other switch” {mqtt=”<[homebroker:scargill/switch/some:state:default]"}

    The scargill.persist file in the persistence directory:
    // persistence strategies have a name and a definition and are referred to in the "Items" section
    Strategies {
    everyFive : "0 0/5 * * * ?"
    everyTen : "0 0/10 * * * ?"
    everyFifteen: "0 0/15 * * * ?"
    everyDay : "0 0 0 * * ?"

    // if no strategy is specified for an item entry below, the default list will be used
    default = everyChange
    }

    /*
    * Each line in this section defines for which item(s) which strategy(ies) should be applied.
    * You can list single items, use "*" for all items or "groupitem*" for all members of a group
    * item (excl. the group item itself).
    */
    Items {
    Azimuth, Elevation : strategy = everyFive
    }

    Now your rule to do something during Sunrise:

    rule "Example Rule at sunrise"
    when
    Item Sunrise_Event received update ON
    then
    …// do something for example change value of some switch
    sendCommand(SomeSwitch, ON)
    end

    Is that all? NO. You have to put some configuration stuff in openhab.cfg too for the Astro and MQTT bindings (your location, MQTT names, addresses, etc.)…

  9. Thanks Peter, very useful. BTW, I’m also around your vintage (1955), but sadly, a Unix user since 1981 and a southerner (London). The ESP is a fantastic discovery for me, I’ll be tracking the Lua and other software avidly, thanks again.

    • You are indeed around my vintage – and you sitting at your ECLIPSE environment and me doing the same in Windows in Eclipse – it’s probably the same 🙂 All very exciting stuff – have fun. Keep an eye on MQTT – I think that is the more important development as it offers a level of security missing from IOT up to now perhaps?

  10. So much appreciate your hard work on this. But, I am confused about where best a simple arduino user learns to softserial the correct “AT + whatever” codes to the “proper” (best: cheapest, reliable, most recent) ESP8266 module to datalog to the web. Can you share some insight on this?

  11. If you must use the AT set (one would assume an Arduino user understands C and the C here is not that much different) then you will need to consider watching the serial input – checking for the correct responses and moving on. Simply putting in delays is likely to catch you out with “busy” signals at some point – and will definitely slow things down. Use an interactive terminal to see what is going back and forth so you know what to look for. ESPLORER (Google it) is great for this – links for that are on the blog. Start off with something simple like the ESP-01 – ensure you have a reliable source of power 3v3 at more than 250ma (the tiny regulators on some Arduinos are absolutely no use for powering ESP8266 boards – and make sure that if you use a 5v Arduino, you level shift the serial input to the ESP-01 – the simplest way is with resistors. If you can, it is better to use 3v3 throughout then you don’t have to worry about that. While the ESP-01 is the easiest to use it also has the least number of pins. ESP-03 and ESP-12 have lots of pins but are not quite so convenient for prototyping. I’m expecting a brand new board turning up in the next few days with some pretty good prototyping kit built in – a full blog will emerge as soon as the hardware turns up and I get a chance to experiment. Check in here next week for more.

  12. As much as I hate Windows, I have to appreciate the fact that some ambitious (and much younger) folks are busy porting most of the Linux tools to it. So if you have a favorite (open s ource) tool, there is a good chance that it will run on both boxes.
    I have a long, frustrating history with home automation, and this little module may just solve the problem of controlling a 10 cent light with a 50 dollar controller. Keep up the good work.

    • Hah! Well let me assure you I’m not “much younger” at 60 and give me Windows any day.. and your comment is bang on – I have been involved in home automation since the 80s and I just cannot see the point of the 50 dollar controller. So the way things stand if I can just get the last part of my MQTT controller working – I expect to have a board+psu costing no more than £9 altogether – controlling a light (which in the UK, a modern LED light or CFC is likely to cost maybe £3 – so the lamp to controller ratio of 3:1 or better and that includes temperature sensing or even temperature+humidity. Almost there.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s