{"id":349,"date":"2018-07-03T12:45:50","date_gmt":"2018-07-03T16:45:50","guid":{"rendered":"http:\/\/www.thesmarthomehookup.com\/test_install\/?p=349"},"modified":"2023-02-08T17:11:47","modified_gmt":"2023-02-08T22:11:47","slug":"understanding-mqtt-how-smart-home-devices-communicate","status":"publish","type":"post","link":"http:\/\/www.thesmarthomehookup.com\/test_install\/understanding-mqtt-how-smart-home-devices-communicate\/","title":{"rendered":"Understanding MQTT: How Smart Home Devices Communicate"},"content":{"rendered":"<p><iframe width=\"560\" height=\"315\" src=\"https:\/\/www.youtube.com\/embed\/NjKK5ab0-Kk\" frameborder=\"0\" allow=\"autoplay; encrypted-media\" allowfullscreen=\"\"><\/iframe><\/p>\n<p><strong>What is MQTT?<\/strong><\/p>\n<p>Today on the hookup we\u2019re going to take a detailed look at the lightweight internet of things messaging platform known as MQTT.<\/p>\n<p>MQTT is a lightweight messaging protocol that allows your smart home devices to communicate with home assistant, or any other mqtt broker.&nbsp; Hassio has mosquito mqtt broker as an addon, and I highly recommend everyone running home assistant also has an MQTT broker running.&nbsp; You can use remote brokers like cloudMQTT, but I really try to handle everything in house so I\u2019m not reliant on cloud services for my smart home functionality.&nbsp; Also, because everything in my smart home is within my network there\u2019s no reason to do any port forwarding to expose my MQTT broker, or my specific devices, to the rest of the internet, giving me a little extra peace of mind.<\/p>\n<p>Each MQTT message contains 3 parts, the message, the topic, and the quality of service (or QoS) packet, well talk about the QoS part later, for now lets say I want to turn on my hallway light.&nbsp; I would publish the message \u201cON\u201d to the topic \/lights\/hallway.&nbsp; This process involves 3 steps.&nbsp; First, I use a client like Node-RED or Home Assistant to publish my message, this message is sent to my MQTT broker.&nbsp; That broker will then post my message to that specific topic, and my hallway lights, which are subscribed to the topic \/lights\/hallway\/ will receive the message and turn on.<\/p>\n<p>Your MQTT broker knows a lot about its clients.&nbsp; An MQTT broker knows which clients have connected to it in the past and remembers them by their \u201cClientID\u201d.&nbsp; It also knows which clients are subscribed to which topics.&nbsp; It also knows which clients are currently connected because those clients will send a \u201ckeep alive\u201d message to the broker at specific time intervals.&nbsp; The keep alive timeout is set on the MQTT broker and by default is 60 seconds on Mosquito, if a client goes 60 seconds without sending an MQTT message mosquito will ping that client\u2019s IP address to see if it is still connected.<\/p>\n<p>It\u2019s important for an MQTT broker to know if a client is connected, this is because each client can also have a birth message and a last will and testament message.&nbsp; The birth message will be published every time a client reconnects to the broker, this is a neat feature, but nothing mind blowing.&nbsp; The last will and testament message is a lot cooler.&nbsp; Using a LWT message you can have a device alert you that it has gone offline.&nbsp; How does a device alert you if it is offline?&nbsp; The MQTT broker stores a LWT message for each client that connects to it, and when that client disconnects the broker will then publish that message to the specified topic.<\/p>\n<p>Lets take a look at one of my favorite MQTT clients, node-red. In node-red you\u2019ve got two major nodes relating to MQTT, one is the publish node and the other is the subscribe node.&nbsp; Lets start with the publish node.&nbsp; In this node you can send any message you\u2019d like to any topic you\u2019d like.&nbsp; For testing out specific payloads it\u2019s useful to pair this with an inject node and then select \u201cstring\u201d from the dropdown menu. Within the publish node you\u2019ll see a few options. The first is QoS, this stands for quality of service, which is along with total message size is a huge advantage of using MQTT over something like the HTTP Get or post method.<\/p>\n<p>Quality of Service has 3 levels, QoS 0 is often called fire and forget, lets say my wife has tasked me with getting my daughter to clean her room.&nbsp; QoS 0 would be like blindly saying \u201cScarlett go clean your room\u201d, where I don\u2019t really worry about whether she heard me, I said it and that\u2019s enough.&nbsp; QoS 1 is a bit different, and is generally referred to as \u201cat least once\u201d, this would be like me saying \u201cScarlett go clean your room\u201d, and then continuing to say that same phrase every 20 or so seconds until I HEAR her respond with \u201cokay Dad, I heard you\u201d, the problem with this that if she\u2019s responding, and I can\u2019t hear her respond I will continue to say \u201cclean your room\u201d and she will think I am telling her to clean her room again even after she has started to clean her room.&nbsp; QoS2 is the final and most data heavy publish.&nbsp; It is known as \u201cexactly once\u201d.&nbsp; This means that I say \u201cScarlett go clean your room\u201d, she says \u201cI heard you\u201d, I cross telling her off of my todo list and say to her \u201cokay, I\u2019m done telling you\u201d, and she says \u201cthanks for this wonderful conversation\u201d which means the transaction is complete.&nbsp; Lets say I didn\u2019t hear her say \u201cI heard you\u201d, and I said \u201cscarlett go clean your room again\u201d, she would know that because I didn\u2019t say \u201cokay I\u2019m done telling you\u201d in between those messages that I must not have heard her the first time and that I am not telling her to clean her room again, I\u2019m just repeating the first message.<\/p>\n<p>In most home automation cases, QoS 2 makes the most sense, because we\u2019re not really worried about the amount of data we need to transfer and a few extra bytes here and there won\u2019t even be noticeable compared to the rest of our network traffic.&nbsp; In addition to QoS you can also choose to send a retain flag.&nbsp; The difference between QoS and the retain flag is that QoS only applies to currently connected clients.&nbsp; If a client is offline during a publish it will never receive the message.&nbsp; A retain flag tells your MQTT broker (mosquito in my case) to hold onto the last message in each topic in case a new client subscribes to it.<\/p>\n<p>Let me show you the difference using node red.<\/p>\n<p>First, I\u2019ll pair an MQTT send node with an inject node and select String for my inject payload.&nbsp; Lets call this payload \u201cnot retained\u201d you can also see that there is an option to set a topic in the inject node, you can also set the topic in the MQTT send node, which is what I\u2019m going to do so we\u2019ll just leave this blank.&nbsp; Next I\u2019m going to jump over to the send node and select QoS 2 and false for the retain flag.&nbsp; I\u2019m also going to pair an MQTT subscribe node with a debug node so we can see what\u2019s would be sent to our internet of things devices.&nbsp; I\u2019ll set the topic to mqtt_test and the QoS to 2.&nbsp; Before I deploy this, I\u2019m going to cut this subscribe node out so we aren\u2019t actually connected.<\/p>\n<p>&nbsp;<\/p>\n<p>Lets deploy and send our message.<\/p>\n<p>As expected, we can\u2019t see anything in our debug window because we are not subscribed to the topic.&nbsp; Lets paste our subscribe sequence back in and deploy it.&nbsp; As expected we still don\u2019t see anything because we sent our mqtt message while the client was not subscribed to the topic.<\/p>\n<p>Now lets look at a retained message.&nbsp; I\u2019ll change my inject node string to \u201cretained\u201d and I\u2019ll change my mqtt publish node retain flag from \u201cfalse\u201d to \u201ctrue\u201d.&nbsp; I\u2019m going to cut out my subscribe node again and deploy.&nbsp; Now we can send our retained message.<\/p>\n<p>As expected, we still can\u2019t see anything in our debug window because we aren\u2019t subscribed to that topic, but here\u2019s where the interesting thing happens, lets paste our subscribe sequence back in and deploy.&nbsp; As soon as we resubscribe the last message in the topic is sent to us.&nbsp; Lets cut that sequence out again, deploy to unsubscribe, and then paste it back it and deploy to resubscribe.&nbsp; You can see the message got sent again, it will continue to be sent to any client that connects to that topic until another retained message is sent.&nbsp; This can sometimes cause an issue.&nbsp; Lets say I\u2019ve misconfigured some of my devices and some of them send retained messages and some of them don\u2019t.&nbsp; I\u2019m going to send the message \u201cnot retained\u201d to this topic with a false retain flag.&nbsp; Lets send it while we are subscribed, as expected it comes through no problem.&nbsp; But lets simulate a device disconnecting and reconnecting.&nbsp; You can see that instead of the last message sent to the topic, which was \u201cnot retained\u201d we instead got the message \u201cretained\u201d.&nbsp; This could cause a real issue if your device needed to have the \u201cnot retained\u201d message to work as intended.<\/p>\n<p>The only way to clear a retained message from a topic is to send a blank retained message to that topic.&nbsp; We\u2019ll just clear out our string, select true for our retain flag and send it out.&nbsp; Now you can see that my simulated device disconnect and reconnect results in no message being sent.<\/p>\n<p>&nbsp;<\/p>\n<p>Maybe you\u2019ve done your homework and were under the impression that QoS 2 would ensure delivery of messages even to offline clients.&nbsp; This does work, but only under a few conditions that are not all that common in IoT devices: number 1. The client must have previously been connected to the broker so that it\u2019s ClientID and subscribed topics were stored.&nbsp; And number 2. The client must have connected with what is called a non-clean or persistent session.<\/p>\n<p>The most popular mqtt library used in IoT projects is the pubsubclient, which is a great library made by Nick O\u2019leary, the same guy that created node-red, doesn\u2019t support persistent sessions, which means every IoT device that uses pubsubclient (which includes all of your tasmota devices) do not support QoS2 persistance, so you\u2019re much better off using the retain flag if you want those messages to be delivered to offline clients.<\/p>\n<p>&nbsp;<\/p>\n<p>The last thing I want to talk about is my MQTT topic schema.&nbsp; If you\u2019ve followed my channel you have seen projects where I use both schema types.&nbsp; In my wireless doorbell video I published the state of the front door to doors\/front where I put the topic type first, and the specific device second.&nbsp; But in my Roomba control video the topics were Roomba\/commands, Roomba\/status and Roomba\/charging.<\/p>\n<p>&nbsp;<\/p>\n<p>Having a consistent MQTT schema lets you debug your messages much more easily using topic wildcards.&nbsp; Lets head back over to node red and look at how to properly use wildcards.&nbsp; For the window sensors in my house, each of them sends a retained closed message to the topic windowStatus slash and then the specific window name.&nbsp; By subscribing to the topic windowStatus slash pound sign, it will act as a multi-level wildcard that picks up messages from each window, and since they are all retained messages they all come through at the same time.<\/p>\n<p>&nbsp;<\/p>\n<p>Using the other schema allows you to isolate a specific device and see all of the mqtt messages send from and to that device.&nbsp; By subscribing to Roomba slash pound sign I can see any information sent from that specific device.&nbsp; It\u2019s up to you which schema you want to use, but I recommend you pick a single schema and stick with it (unlike me).&nbsp; If I had to pick one I\u2019d say defining the device first and then the topic subject second is the superior method.<\/p>\n<p>&nbsp;<\/p>\n<p>Using this method for my windows I would define a single level wildcard using the + symbol.&nbsp; So to subscribe to familyRoom\/windowStatus and kitchen\/windowStatus I\u2019d use the wildcard topic +\/windowStatus<\/p>\n<p>&nbsp;<\/p>\n<p>Understanding MQTT will allow you to make interesting programming decisions when writing programs for your sensors.&nbsp; If I knew then what I know now I would have done a few things differently when programming some of my smart home devices, but that\u2019s all part of the fun of it I guess! If you have any other questions about MQTT, or you\u2019ve got a suggestion for another video about MQTT in home automation leave them down in the comments.&nbsp; If you enjoyed this video please consider subscribing, and as always, thanks for watching the hookup.<\/p>\n<h3>Links<\/h3>\n<p>Follow me on Twitter: @TheHookUp1<br \/>\nSupport my channel:<br \/>\nPatreon:&nbsp;<a href=\"https:\/\/www.patreon.com\/thehookup\">https:\/\/www.patreon.com\/thehookup<\/a><\/p>\n<p>Music by www.BenSound.com<\/p>\n","protected":false},"excerpt":{"rendered":"<p>What is MQTT? Today on the hookup we\u2019re going to take a detailed look at the lightweight internet of things messaging platform known as MQTT. MQTT is a lightweight messaging protocol that allows your smart home devices to communicate with home assistant, or any other mqtt broker.&nbsp; Hassio has mosquito mqtt broker as an addon, [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":2605,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[5],"tags":[],"class_list":["post-349","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-tutorials"],"acf":[],"mb":[],"mfb_rest_fields":["title","gutenberg_elementor_mode"],"_links":{"self":[{"href":"http:\/\/www.thesmarthomehookup.com\/test_install\/wp-json\/wp\/v2\/posts\/349","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.thesmarthomehookup.com\/test_install\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.thesmarthomehookup.com\/test_install\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.thesmarthomehookup.com\/test_install\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/www.thesmarthomehookup.com\/test_install\/wp-json\/wp\/v2\/comments?post=349"}],"version-history":[{"count":4,"href":"http:\/\/www.thesmarthomehookup.com\/test_install\/wp-json\/wp\/v2\/posts\/349\/revisions"}],"predecessor-version":[{"id":2360,"href":"http:\/\/www.thesmarthomehookup.com\/test_install\/wp-json\/wp\/v2\/posts\/349\/revisions\/2360"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/www.thesmarthomehookup.com\/test_install\/wp-json\/wp\/v2\/media\/2605"}],"wp:attachment":[{"href":"http:\/\/www.thesmarthomehookup.com\/test_install\/wp-json\/wp\/v2\/media?parent=349"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.thesmarthomehookup.com\/test_install\/wp-json\/wp\/v2\/categories?post=349"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.thesmarthomehookup.com\/test_install\/wp-json\/wp\/v2\/tags?post=349"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}