MQTT topic tree structure improvements

I’ve been unhappy with my old MQTT topic tree structure for quite some time now and did some thinking/research on how to improve it. Google pointed me to Tinkerman’s article on a similar subject, and I started planning on improvements.

I will repeat a quote I used in my last post “If you can’t measure it, you can’t improve it”, only this time re-phrase it as “If you can’t visualize it, you can’t improve it”. Yes, my first effort would be to visualize my MQTT topic tree and the rest should be fairly easy.

Ben Hardill had done some excellent work on the subject and his D3 MQTT topic tree visualizer is exactly what I needed. It was quite tricky to get it to work. I wanted to use the new Mosquitto MQTT support for websockets, so I had to upgrade to that new version. Websocket support must be explicitly enabled when building Mosquitto, thankfully Jeremy Gooch has created *excellent* instructions for doing that, I could not have done it without these.

Finally, I had to modify Ben’s mqttws31.js file to work with Mosquitto 1.4, my fixed version is on Github.

With the visualization working, I did about 1003 iterations to get to the following structure, still lots of nodes to add and in rough shape:

 

The root contains three folders to group my connected things into the categories “places”, “cloud” and “people”. I plan to add more categories there, for example, “objects” to track vehicles.

The “places” will contain places and properties that are part of my IoT home automation system, starting with “our place”:

 

“Our place” is a house with garden, so I broke this in two: the “house” and “outside”. The “house” is branched into three sub-categories: “common”,”first floor” and “second floor”. The “common” section will contain common components that can’t be grouped otherwise. The “outside” will obviously group anything in the garden like outside conditions, greenhouse, front door etc (all WIP)

The “Common” category currently has these leaf nodes:

 

 

The “raspberry pi” branch expanded:

 

The “rfm2pi” leaf represents the rfm2pi board that handles the wireless transmissions, it has incoming and outgoing leaves. The “out” topic is used to transmit, I use it to send time packets every 5 minutes for example and also to send control packets to remote nodes. Node-RED subscribes to the different NodeID leaves in the “in” topic to handle the raw packets of data and further process it, feeding the other branches with processed data.

The other leaves in this branch are:

 

The “power monitor” node data is provided by my sensing shield project, still working nicely.

The “solar” node data is provided by the hot water tank controller I described here, the firmware has changed notably from then, but the hardware is same. Note that it is a bi-directional node with “in” and “out” leaves because it can control the heater element in the water tank as well should the sunshine be less than needed to heat up the water.

The “doorbell” node is fed by my IoT doorbell project.

The “security system” node is fed by my home security system interface project.

The “heat pump” node is still work in progress, I plan to have both “in” and “out” sub-topics as it can both control and provide information about the heat pump state. Using my Daikin controller project there.

 

The “first-floor” branch has the following structure:

 

 

My friend Glyn Hudson of OpenEnergyMonitor has sent me an EmonGLCD unit as a gift, this is the data provider for temperature in the “kitchen” node.

I have a fire sensor alarm project that is feeding the “fire alarm” node in the “kitchen” branch.

The house’s door, window, and PIR states are fed by my security system interface project. That data helps the central heating system, I plan to stop the room thermostats if a window is open.

Living room temperature is provided by a Funky+DS18b20 project.

TV status is provided by the device tracking project. I also can control its state using the LG TV controller project.

 

 

“Second-floor” branch has lots of to-dos, but in its current shape looks like this:

 

Room temperature data providers are  Funky+DS18b20 project.

PS3 game console status is fed by the device tracking project.

 

“Outside” branch looks like this, again I do have still more branches to add:

 

 

I have two software bots to feed WeatherUnderground’s forecast and current weather data, quite practical since I do not have rain/pressure/wind sensors available. I use the forecast data to send myself frost alerts and prepare the garden for such events. Still need to blog about these, as I am doing all this in Node-RED now.

The Funky+DHT22 project is the data provider for outside temperature/humidity. That data is being enriched by adding calculated dew point as well.

 

The “cloud” branch is naturally grouping all cloud-connected “things” like EmonCMS, Pushbullet, Twitter, email, Twilio etc. Still WIP.

 

Node-RED flows watch over the emoncms/out topic and send that data to EmonCMS.

The twitter/out topic is what gets published to my house’s Twitter account. Follow it 🙂

 

 

..and finally the “people” branch:

 

Presence detection project feeds my and wife’s presence at home topics.

Weight is being tracked with my keypad project.

My location is being tracked with OwnTracks, still, need to blog about that project.

 

That’s roughly where I stand now on my MQTT topic tree restructuring project. I enjoy looking at the D3 visualizer, it is animated and data flies in as it arrives, very beautiful indeed. I like to think of it as my house’s central nervous system visualized.