ESP8266 MQTT Publications

The PlanIn my last blog I referred to publishing temperature etc and there is one issue with the existing MQTT library that makes that an issue: The MQTT publications are not queued.

For the benefit of anyone still not into the MQTT thing – by publications I mean “sending a message” comprising a topic and a message. The topic might be “BathroomTemperatureReading” and the message might be “20.6”.

What happens right now – and Tuan has said he’ll fix this… if you send 2 messages out in a row IMMEDIATELY after each other – only the last one will go. Well, that’s fine unless you have to send one message off to a thermometer display and another message off to a humidity display.

So… this evening I wrote a queue. This is possible because of a “callback routine”. When you fire a message off it is handled in the background and when it’s done it calls an empty routine. I’ve trapped that.. the basic idea is this.  

Clear a “done” flag at the start and from there on..

If the done flag is clear, set it and send out a publication. If it is NOT clear, put the message in a queue.

In the callback routine, clear the flag – check a queue, if there is anything to go – send out the publication.

Simple really.

So here’s the code I’ve added to the start of the MQTT package..

#define QUEMAX 10  // 10 messages -  need more? – increase
#define QUESIZE 50 // my topics and messages are never more than 50 chars
struct Queues{
    MQTT_Client* cli;
    char top[QUESIZE];
    char mes[QUESIZE];

struct Queues myQueue[QUEMAX];

int queuein=0;
int queueout=0;
int isPublished=0; // my flag to see if I can publish or need to put in a queue

All simple enough – the queue can be maximum 10 deep (it’s rotary – when I get to the end I roll around to the start). I save the client pointer, the topic text and the message text in an array of simple structs.

At the beginning both the in pointer and out pointers for the queue are cleared and isPublished is clear.

Instead of the main publish routine – I have my own – which checks the flag before sending off a message.

void petesPub(MQTT_Client* cli, char *top, char *mes)
if (isPublished==0)    { isPublished=1; MQTT_Publish(cli, top,mes,strlen(mes), 0, 0); }
    if (++queuein>=QUEMAX) queuein=0;

You can see the message going in the queue unless the flag is clear… and in the callback routine, called when a message has gone…

void mqttPublishedCb(uint32_t *args)
    MQTT_Client* client = (MQTT_Client*)args;
    INFO("MQTT: Published\r\n");
    if (queuein!=queueout)
        if (++queueout>=QUEMAX) queueout=0;

The first two lines were there already – I don’t think I actually need to store the client info – I think it gets passed – however – what I have works for now.

So you can see I clear the published flag, look to see if there is anything in the queue and if there is – publish it. I’m sure there’s a bit of recursion going on there – but it all works.

And that’s me done for the day. Meetings again this week so I’ll not get much more done now until the weekend but DHT22, RTC and message queue – not a bad start for the week.

I’m still begging anyone with assembly skills to look at the WS2812B code to see if they can make an accurate job of it Smile

7 thoughts on “ESP8266 MQTT Publications

  1. Don’t want to copy the discussion from, but I still believe that using mqtt publishing queue is not the right way. In your example above you are taking 1040 RAM bytes for the queue.
    If you want to queue temperature results, why don’t you make a small queue of 10 x int16_t values (20 bytes total) for the temperature measurements? Then in the callback you can create the mqtt message for current value from the queue.
    It is questionable how to define those numbers – 10 entries, 50 chars for topic and value? One could optimize this for given situation, surely. But is it user friendly? The depth of 10 translates to a period of time (e.g. 10 measurements every 6s) that could be buffered. This could be related to the maximum “offline” time that the node can handle – 100s. This sounds reasonable – it will survive router reboot fine. But if one changes the polling rate from 10s to 5s than router reboot becomes a problem.
    In fact, imagine the case when you want to run on batteries. You cound design your application to wakeup every 60s. If you use the RTC RAM (256 bytes for the user) you could store there 128 temperature values. So first 127 wakeups you only measure, put in the RTC RAM, and the module goes in sleep immediately (1-2s), without starting the WIFI modem at all. At the end, at 128th wake up it connects and pushes all 128 results at once. At 160MHz you could achieve 10-15s active time (or even less) every 128 minutes. This translates to more than a year life time with alcaline batteries.

    I think that the queue has an issue – you have to store all arguments that are needed for the publishing message in the mqtt task. Now you have topic and data, but at some point you might need also the QoS (not supported in Tuan’s code but easy to add). I am missing the “retain” flag too.

    • Hi – my answer to the use of 1k of RAM is “so what” – there is around 30k available right now. Why do I want a queue? – because I want to send results to different systems at the same time. I want to send the temperature to an OpenHab thermometer and the humidity elsewhere. I also want to send the information in a different format to a wall-mounted MQTT client. Why on earth would I want to make it more complicated than that 🙂 In tests sending off to 3 different locations occurs visually instantly.

      User-friendly? I don’t care. I’ve selected 50 bytes for me – someone else can make it whatever they want. With a little care one could malloc the space and make it completely adjustable – I wanted a quick fix and with plenty of RAM available and precious little else to do with it – why not. I’m not doing any polling at all. It’s all done under the callbacks so that in the background the instant one message is complete – any in the queue are processed. As for speed of messages, I’ve measured that I can send around 1000 messages a second using MQTT and Mosquitto so all of this is taking a tiny proportion of available time and resources.

      I don’t want to run on batteries and have no intention of putting stuff to sleep which would change the game entirely. I will need some outdoor stuff and they will use a single Lithium battery – these are readily extracted from old laptops and at 3v6 are ideal for the job. Solar panels today are dirt cheap – 90p for a Lithium charger and a few quid for a cell capable of keeping the Lithium in good shape. This morning I added (at the cost of 2 bytes per item) the retain and QOS. I think that TUANs code DOES support these but I would not start an argument as I’ve not tested it. It is essential incidentally to have the latest version of Tuan’s code as of a few days ago otherwise the publication callback does not work.

      Thanks for the feedback, others may have different power requirements and it’s always good to see other ways to do things.

      • Sure this fits your case, but I wanted to point those issues because of the pulicity of the blog and readers that might benefit from knowing them.
        As far as I know Tuan’s code, sending to different brokers will require setting up several MQTT client instances, each with own buffers (queues), each with own task (MQTT_Task), each with own task queue, each with its own connection. Tuan’s code is not prepared for this – task priority is fixed for example. So going in this direction is questionable.
        I would imagine use case where you might have a list of multiple brokers to work with and switch them if one is unavailable. But still DNS resolving could bring back the currently active “”. But rest of the things are better managed with MQTT bridges and subcribers. Your “Wall-mounted MQTT client” is not a broker, so you cannot send (mqtt) directly to it – this is going to be done as your “wall mounted mqtt client” subscribes to humidity topic on the single broker. This is THE PATTERN that MQTT employs – the publish-subscribe one.
        About the “only 1K ram” – yes, this looks fine in current context, but imagine if Tuan accepts the request for built in queues and integrates them. Then NodeMCU and/or NodeLua take the open source code of Tuan. So, already limited Lua heap is reduced more.
        In some cases (patterns) your node gets “base URI” from somewhere to prepend its own “relative” topics – e.g. base URI is /global/company1/building5/floor8/room231/rack5 and the node adds its /device887545637/reading/temperature to get the final “/global/company1/building5/floor8/room231/rack5/reading/temperature.
        To manage this one should be aware in design time about possible lengths of the base URI. And this blows as 10 entries * 3 brokers * 50 bytes overhead – while it could be single one.
        I mean that if it can be saved, it should be saved.
        And I don’t see how adding queues (with design-time and run-time considerations) could be called “not complicated”. My experience says that OS-style of “simplifying” things, applied to an embedded system, only adds up more troubles and wastes resources (like the buffers here).
        Espressif had done a great job in providing event based SDK, or framework (based on LwIP event API). Converting this to old-school OS threaded style is not going to simplify the things.

        And if don’t care about sleep, be prepared to change your solar panels to something that can supply steady 70mA, day and night, all seasons. Dimensions of such panel is going to be something like 10″x10″ (or more), so it can give 200mA during the day to charge your li-ion accumulators. Low quality one will give up after 300-500 cycles, or even less if they came from old notebook pack.

        For retain and QoS, yes, the MQTT core in Tuan’s code supports them but those arguments were not added to the API.

    • Gicho – Re: 8.45am comments (why on EARTH WordPress doesn’t have a reply box below the last comment escapes me) – as there are currently no viable MQTT alternatives to Tuan’s code (the only other project I’m aware of is 2 months old and doesn’t work) I think we need to be generous – he’s done a cracking job. If you could not get through to one broker – you’d try another – so you don’t need separate queues for each one. It would be easy to check the callback and if that broker failed just go off looking for another – I plan to do the same with the access points.

      My wall mounted client is a client and of course I can send messages to it (transparently via the broker) – that’s the whole point and trhe reason that MQTT is a better choice than the various alternatives, trying to make WIFI networks etc. This just works.

      If Tuan accepts the request for queues we can always turn them off or just keep using the existing code. I spent quite some time with Lua and I don’t really think the chip has enough RAM for really serious size project – hence taking the plunge and learning how to compile in Windows… there’s enough space for quite big projects in C and the Windows environment is not THAT far off what people are used to with Arduino once set up properly.

      Incidentally I see absolutely no reason for the very long topic names you mention. As noone else but me is using Mosquitto (and others like RABBITMQ can be configured to give each client their own private space) I plan on using topics such as “livingroom/lamp5”

      If we’re going to have a conversation please avoid slightly derogatory terms like “Converting this to old-school OS threaded style is not going to simplify the things”. I write software the way I feel comfortable – I don’t work for an organisation that forces me to do anything else – and as much of this is done by hobbyists I’d imagine many feel the same way. The fact it today I have an RTC and queing which Tuan himself accepts is necessary.. yesterday I didn’t – old-school works for me. I do enjoy differing views but yours is also only one view.

      Batteries I would never use poor quality ones – I’ve a vast experience of throwing cheap rechargeables in the bin after less than a year – false economy, Your point about 70ma is valid but in my case – it’s not that big a deal. I made the sensible move when we first came here of running a hefty 12v lead around the back of the entire garden – didn’t know it then but it’s been awfully handy since as typical garden solar lights are not worth “a light” with one notable exception from Maplins.. Not sure where you get 10″ from – For under £8 Ebay have 500ma (British weather lets assume 300ma or so) at 6v size 6″ * 8″. Suitably low enough current not to fry the charger, high enough to keep the battery in good condition I would have thought.

      Erm that last point – do you want to clarify that – has he actually implemented QOS or not? I’ve used it with MQTT-SPY and MOSQUITTO but not yet with the little boards – those extra features are pretty mandatory if you wanted to switch things on and off periodically.

    • Sorry, not planned to make it a fight. But I am biased in this area and probably I give too much passion into it – tired of fixing “simplified” solutions at my day job.
      I just wanted to add some value to this blog topic.

      Those solar panels are rated at max solar radiation (1400w/sqm). In winter time you will get maybe 10 cloudy and/or foggy days in a row, with something like 60-80W/sqm – radiation. Take the 10-15% efficiency into account, calculate that you have 8 hours at best and 16 hour darkness, take that li-ion don’t like low temperatures. Average for a 24h, you will get 1/30th of the rated power. So probably we are speaking about >2000mA panel to fit the bill.
      I am not saying that this solution is not going to work. I just want to give some real calculations. So anyone could see if this kind of solution is suitable.
      Also, in some cases it makes a lot of sense to be as fast as possible. If you want to check what to dress on, than 1 hour update might be sufficient.
      But imagine that you want to observe and analyze the behavior of the heatpump in your house. In this case you would want to get a new value as fast as possible, probably faster that 1s. This is very helpful when you want to configure and optimize the controller (PID or whatever). And this does not conflict with “no buffers” event solution.

      The huge difference is the place where you put the queue and the size of the content that you put on it.

      About altenative MQTT implementation – I have ported the code from here:
      to the ESP8266. It works but I had not enough time to polish it (the port) before I saw Tuan’s implementation (I think his one is based on eclipse paho).

  2. That’s fine Gicho – do remember that for many this is not a job – it is supposed to be fun – so I tend to come down hard on people who appear to be lecturing. This has to be a friendly place. Your points about solar are indeed valid, thanks for taking the time to do the calcs. I must admit I should have done the same given the rubbish weather we’ve had in the Northeast of England for, oh, the last 20 years. I think your 10*10 might just be about right – once I get the code sorted it is going to get the test.

    The alternative implementation – are you planning to continue on with it??? – it’s always possible that Tuan might concentrate on building his into Lua in which case a continually developed alternative will have a good place. I certainly am glad I discovered MQTT – best thing that’s happened to my efforts to date.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s