Understanding MQTT: How Smart Home Devices Communicate

What is MQTT?

Today on the hookup we’re going to take a detailed look at the lightweight internet of things messaging platform known as MQTT.

MQTT is a lightweight messaging protocol that allows your smart home devices to communicate with home assistant, or any other mqtt broker.  Hassio has mosquito mqtt broker as an addon, and I highly recommend everyone running home assistant also has an MQTT broker running.  You can use remote brokers like cloudMQTT, but I really try to handle everything in house so I’m not reliant on cloud services for my smart home functionality.  Also, because everything in my smart home is within my network there’s no reason to do any port forwarding to expose my MQTT broker, or my specific devices, to the rest of the internet, giving me a little extra peace of mind.

Each MQTT message contains 3 parts, the message, the topic, and the quality of service (or QoS) packet, well talk about the QoS part later, for now lets say I want to turn on my hallway light.  I would publish the message “ON” to the topic /lights/hallway.  This process involves 3 steps.  First, I use a client like Node-RED or Home Assistant to publish my message, this message is sent to my MQTT broker.  That broker will then post my message to that specific topic, and my hallway lights, which are subscribed to the topic /lights/hallway/ will receive the message and turn on.

Your MQTT broker knows a lot about its clients.  An MQTT broker knows which clients have connected to it in the past and remembers them by their “ClientID”.  It also knows which clients are subscribed to which topics.  It also knows which clients are currently connected because those clients will send a “keep alive” message to the broker at specific time intervals.  The keep alive timeout is set on the MQTT broker and by default is 60 seconds on Mosquito, if a client goes 60 seconds without sending an MQTT message mosquito will ping that client’s IP address to see if it is still connected.

It’s important for an MQTT broker to know if a client is connected, this is because each client can also have a birth message and a last will and testament message.  The birth message will be published every time a client reconnects to the broker, this is a neat feature, but nothing mind blowing.  The last will and testament message is a lot cooler.  Using a LWT message you can have a device alert you that it has gone offline.  How does a device alert you if it is offline?  The MQTT broker stores a LWT message for each client that connects to it, and when that client disconnects the broker will then publish that message to the specified topic.

Lets take a look at one of my favorite MQTT clients, node-red. In node-red you’ve got two major nodes relating to MQTT, one is the publish node and the other is the subscribe node.  Lets start with the publish node.  In this node you can send any message you’d like to any topic you’d like.  For testing out specific payloads it’s useful to pair this with an inject node and then select “string” from the dropdown menu. Within the publish node you’ll see a few options. The first is QoS, this stands for quality of service, which is along with total message size is a huge advantage of using MQTT over something like the HTTP Get or post method.

Quality of Service has 3 levels, QoS 0 is often called fire and forget, lets say my wife has tasked me with getting my daughter to clean her room.  QoS 0 would be like blindly saying “Scarlett go clean your room”, where I don’t really worry about whether she heard me, I said it and that’s enough.  QoS 1 is a bit different, and is generally referred to as “at least once”, this would be like me saying “Scarlett go clean your room”, and then continuing to say that same phrase every 20 or so seconds until I HEAR her respond with “okay Dad, I heard you”, the problem with this that if she’s responding, and I can’t hear her respond I will continue to say “clean your room” and she will think I am telling her to clean her room again even after she has started to clean her room.  QoS2 is the final and most data heavy publish.  It is known as “exactly once”.  This means that I say “Scarlett go clean your room”, she says “I heard you”, I cross telling her off of my todo list and say to her “okay, I’m done telling you”, and she says “thanks for this wonderful conversation” which means the transaction is complete.  Lets say I didn’t hear her say “I heard you”, and I said “scarlett go clean your room again”, she would know that because I didn’t say “okay I’m done telling you” in between those messages that I must not have heard her the first time and that I am not telling her to clean her room again, I’m just repeating the first message.

In most home automation cases, QoS 2 makes the most sense, because we’re not really worried about the amount of data we need to transfer and a few extra bytes here and there won’t even be noticeable compared to the rest of our network traffic.  In addition to QoS you can also choose to send a retain flag.  The difference between QoS and the retain flag is that QoS only applies to currently connected clients.  If a client is offline during a publish it will never receive the message.  A retain flag tells your MQTT broker (mosquito in my case) to hold onto the last message in each topic in case a new client subscribes to it.

Let me show you the difference using node red.

First, I’ll pair an MQTT send node with an inject node and select String for my inject payload.  Lets call this payload “not retained” you can also see that there is an option to set a topic in the inject node, you can also set the topic in the MQTT send node, which is what I’m going to do so we’ll just leave this blank.  Next I’m going to jump over to the send node and select QoS 2 and false for the retain flag.  I’m also going to pair an MQTT subscribe node with a debug node so we can see what’s would be sent to our internet of things devices.  I’ll set the topic to mqtt_test and the QoS to 2.  Before I deploy this, I’m going to cut this subscribe node out so we aren’t actually connected.

Lets deploy and send our message.

As expected, we can’t see anything in our debug window because we are not subscribed to the topic.  Lets paste our subscribe sequence back in and deploy it.  As expected we still don’t see anything because we sent our mqtt message while the client was not subscribed to the topic.

Now lets look at a retained message.  I’ll change my inject node string to “retained” and I’ll change my mqtt publish node retain flag from “false” to “true”.  I’m going to cut out my subscribe node again and deploy.  Now we can send our retained message.

As expected, we still can’t see anything in our debug window because we aren’t subscribed to that topic, but here’s where the interesting thing happens, lets paste our subscribe sequence back in and deploy.  As soon as we resubscribe the last message in the topic is sent to us.  Lets cut that sequence out again, deploy to unsubscribe, and then paste it back it and deploy to resubscribe.  You can see the message got sent again, it will continue to be sent to any client that connects to that topic until another retained message is sent.  This can sometimes cause an issue.  Lets say I’ve misconfigured some of my devices and some of them send retained messages and some of them don’t.  I’m going to send the message “not retained” to this topic with a false retain flag.  Lets send it while we are subscribed, as expected it comes through no problem.  But lets simulate a device disconnecting and reconnecting.  You can see that instead of the last message sent to the topic, which was “not retained” we instead got the message “retained”.  This could cause a real issue if your device needed to have the “not retained” message to work as intended.

The only way to clear a retained message from a topic is to send a blank retained message to that topic.  We’ll just clear out our string, select true for our retain flag and send it out.  Now you can see that my simulated device disconnect and reconnect results in no message being sent.

Maybe you’ve done your homework and were under the impression that QoS 2 would ensure delivery of messages even to offline clients.  This does work, but only under a few conditions that are not all that common in IoT devices: number 1. The client must have previously been connected to the broker so that it’s ClientID and subscribed topics were stored.  And number 2. The client must have connected with what is called a non-clean or persistent session.

The most popular mqtt library used in IoT projects is the pubsubclient, which is a great library made by Nick O’leary, the same guy that created node-red, doesn’t support persistent sessions, which means every IoT device that uses pubsubclient (which includes all of your tasmota devices) do not support QoS2 persistance, so you’re much better off using the retain flag if you want those messages to be delivered to offline clients.

The last thing I want to talk about is my MQTT topic schema.  If you’ve followed my channel you have seen projects where I use both schema types.  In my wireless doorbell video I published the state of the front door to doors/front where I put the topic type first, and the specific device second.  But in my Roomba control video the topics were Roomba/commands, Roomba/status and Roomba/charging.

Having a consistent MQTT schema lets you debug your messages much more easily using topic wildcards.  Lets head back over to node red and look at how to properly use wildcards.  For the window sensors in my house, each of them sends a retained closed message to the topic windowStatus slash and then the specific window name.  By subscribing to the topic windowStatus slash pound sign, it will act as a multi-level wildcard that picks up messages from each window, and since they are all retained messages they all come through at the same time.

Using the other schema allows you to isolate a specific device and see all of the mqtt messages send from and to that device.  By subscribing to Roomba slash pound sign I can see any information sent from that specific device.  It’s up to you which schema you want to use, but I recommend you pick a single schema and stick with it (unlike me).  If I had to pick one I’d say defining the device first and then the topic subject second is the superior method.

Using this method for my windows I would define a single level wildcard using the + symbol.  So to subscribe to familyRoom/windowStatus and kitchen/windowStatus I’d use the wildcard topic +/windowStatus

Understanding MQTT will allow you to make interesting programming decisions when writing programs for your sensors.  If I knew then what I know now I would have done a few things differently when programming some of my smart home devices, but that’s all part of the fun of it I guess! If you have any other questions about MQTT, or you’ve got a suggestion for another video about MQTT in home automation leave them down in the comments.  If you enjoyed this video please consider subscribing, and as always, thanks for watching the hookup.

Links

Follow me on Twitter: @TheHookUp1
Support my channel:
Patreon: https://www.patreon.com/thehookup

Music by www.BenSound.com