Tutorial: MQTT and the retain flag

Assumptions and setup

This tutorial assumes:

a single computer (like a Raspberry Pi)
MQTT broker (like Mosquitto) running
the mosquitto_pub and mosquitto_sub commands are available.

The "single computer" assumption avoids having to specify hosts. If you can't meet the "single computer" assumption then simply change all of the mosquitto_pub and mosquitto_sub commands to have a -h «host» parameter where «host» is the domain name or IP address of the host running your MQTT broker.

You may already have Mosquitto (the MQTT broker) installed as part of SensorsIot/IOTstack but you might need to install the command-line tools separately. On a Raspberry Pi, that's:

$ sudo apt install mosquitto-clients

The subscriber in the provider-subscriber model

For any MQTT topic you will normally have a subscriber process running in the background. A Node-Red flow that starts with an "MQTT in" node is a common example of this.

Let's simulate that situation at the command line:

$ mosquitto_sub -t "/test/#" -F "%I %t %p" &
[1] 12750

In words, mosquitto_sub is running in the background (the effect of the & on the end of the command) with job ID [1] (12750 is the process ID). The process is subscribed to any topic string beginning with "/test/". The "#" is a wild-card that spans zero or more levels in a topic string. When it receives a message from the broker, it will print it to the terminal window using the -F format string, which includes the time when the message was received.

The provider in the provider-subscriber model

Now, let's provide a message for that topic. The payload in the following is a JSON string associating the key "d" with the current date and time:

$ mosquitto_pub -t "/test/example" -m "{\"d\": \"$(date)\"}"
2020-09-06T12:35:47+1000 /test/example {"d": "Sun  6 Sep 12:35:47 AEST 2020"}

The second line was printed by mosquitto_sub running in the background. The time at the moment the message was sent to the broker is embedded in the JSON payload on the right hand side. The time when the message was received from the broker by mosquitto_sub is on the left hand side. In other words, getting a message from provider to broker to subscriber is pretty much instantaneous.

You can repeat that test as many times as you like to satisfy yourself that mosquitto_sub will always faithfully echo whatever mosquitto_pub sends.

Experiment 1 – the subscriber stops working

Now, kill the background process, using its job ID:

$ kill %1

This is analogous to Node-Red being taken down. Try sending another message:

$ mosquitto_pub -t "/test/example" -m "{\"d\": \"$(date)\"}"

Silence. In fact, Mosquitto received the message but there were no subscribers so the message didn't go anywhere.

Experiment 2 – the subscriber starts again

Bring up a subscriber again:

$ mosquitto_sub -t "/test/#" -F "%I %t %p" &
[1] 20668

Did it receive the message that was sent to Mosquitto while the subscriber was down? The answer is no. That particular message has been lost.

A scenario

Imagine you are building a light switch. Your hardware is an ESP32, a button, and a relay. Whenever you press the button, the ESP32 toggles the relay and sends an MQTT message to Mosquitto reporting the new state.

From time to time you want to control the light switch from your smart phone. Whenever you launch your browser and connect to the service, you expect the on-screen button to show whether the light is currently on or off.

You want the on-screen button to show whether the light is on or off, even if Mosquitto (the broker) restarts, or you quit and relaunch the browser on your phone or, indeed, if the ESP32 in the light switch restarts after a power failure.

This is where the MQTT "retain" flag can come in handy. What the "retain" flag does is to tell the MQTT broker (Mosquitto) to keep the last message it received for any topic, and always forward that when a new subscriber fires up. Otherwise it behaves the same as a message sent without the retain flag.

Experiment 3 – sending with the retain flag

Our subscriber is still running. Let's send a message with "retain" turned on (the -r at the end of the command):

$ mosquitto_pub -t "/test/example" -m "{\"d\": \"$(date)\"}" -r
2020-09-06T12:57:17+1000 /test/example {"d": "Sun  6 Sep 12:57:17 AEST 2020"}

No difference in the timestamps. Now let's kill the subscriber and restart it:

$ kill %1
$ mosquitto_sub -t "/test/#" -F "%I %t %p" &
[1] 23991
2020-09-06T12:58:34+1000 /test/example {"d": "Sun  6 Sep 12:57:17 AEST 2020"}

Now you can see a difference in the timestamps. The one embedded in the JSON payload is the same as the last time mosquitto_pub was called. The one on the left hand side is when the new mosquitto_sub was launched and re-subscribed to the topic. The broker re-sent the last retained message.

You can repeat the sequence of killing and re-launching the subscriber. It will always receive the most-recent retained message. It doesn't matter if that message is hours, days or weeks out of date.

Even restarting the MQTT broker won't cause a retained message to disappear:

$ docker-compose -f ~/IOTstack/docker-compose.yml restart mosquitto 
Restarting mosquitto ... done
2020-09-06T13:02:25+1000 /test/example {"d": "Sun  6 Sep 12:57:17 AEST 2020"}

What happened was:

Mosquitto (the MQTT broker) terminated, which severed the connection between mosquitto_sub and Mosquitto
The mosquitto_sub process running in the background went into a loop trying to reconnect to Mosquitto
Mosquitto restarted
The mosquitto_sub process running in the background reconnected and re-subscribed to the topic; and
Mosquitto re-sent the retained message.

Experiment 4 – clearing a retained message

So, what do you do if a retained message is so old that it is no longer relevant and you want to clear it?

Simple. Just send another message to the same topic, with an empty payload, and with the retained flag turned on:

$ mosquitto_pub -t "/test/example" -m "" -r
2020-09-06T13:09:09+1000 /test/example

Note that the message with the empty payload was sent to mosquitto_sub running in the background. But that happens exactly once. After relaying the null payload message to all current subscribers, Mosquitto (the MQTT broker) clears the message and won't re-send if any subscriber restarts. Let's prove that:

$ kill %1
$ mosquitto_sub -t "/test/#" -F "%I %t %p" &
[1] 28624

No "empty payload" retained message. Mosquitto (the broker) also won't re-send the null message if it restarts:

$ docker-compose -f ~/IOTstack/docker-compose.yml restart mosquitto
$

Again, no "empty payload" retained message.

But we can be sure the subscriber is still running by sending a non-retained message to the same topic:

$ mosquitto_pub -t "/test/example" -m "{\"d\": \"$(date)\"}"
2020-09-06T13:15:01+1000 /test/example {"d": "Sun  6 Sep 13:15:01 AEST 2020"}

Design considerations

In general, subscriber processes should not try to distinguish between messages according to whether the "retain" flag was set by a provider. However, subscribers need to handle null-payload messages without crashing.

In the case of a smart phone app monitoring a light switch, a null payload message would have the same semantic meaning as no response at all from the broker: the state of the light switch is unknown.

Don't get the idea that the retain flag is useful as a matter of routine. It isn't. It is there to solve very specific problems such as "state" information (eg light switch on or off) that never becomes stale so it's reasonable for it to be provided to subscribers indefinitely.

If your design has a bunch of ESP8266 temperature monitors sending MQTT payloads to Mosquitto which are received by Node-Red, written to InfluxDB, and displayed using Grafana, you really will not need the retain flag. Grafana is always going to fetch the most-recent data from InfluxDB. If you are graphing temperature time-series then gaps in your charts are actually useful information telling you that one or more of your sensors is on the fritz.

Paraphraser/TheRetainFlag.md