Proposal of an MQTT Smarthome 2.0 (Sketch)
Let me introduce a few words that we'll be using in this document:
Datapoint: The semantic description for a certain datapoint, e.g. "hue of a light bulb"
Value: The numerical/boolean/whatever value that is conencted to one datapoint
State: A combination of one or multiple datapoints that are needed to give a representation of a device's state
Topic: The actuall MQTT topic a message will be posted on.
Payload: The actuall MQTT payload of such a message. Datatype can be JSON or String.
Message: Topic with payload
Device: Typically a physical device such as an AVR or a bridge to other home automation systems such as KNX/Homematic.
Service: One device can have multiple services. A service can have exactly one state.
For e.g. an RGB bulb has 3 relevant datapoints to make an overal state:
- hue
- saturation
- brightness
with the according values. In this case it makes no sense to broadcast status messages for a single datapoint per topic, e.g.
datapoint -> value
hue1/status/light/hue -> 1.0
hue1/status/light/sat -> 1.0
hue1/status/light/bri -> 1.0
because to determine the state, all three datapoints are needed.
It is usually not a problem to split these topics, but let's say a script crashes and needs to be restarted. The user then manually changes just the brightness of the bulb. The freshly started script will receive just the brightness and still has no data about the current color of the bulb in a webinterface. Whenever the state of a bulb changes, it makes sense to publish the complete new state, wrapped in a JSON string on one topic:
hue1/status/light -> { "hue":1.0, "sat":1.0, "bri":1.0 }
-> { datapoint1: value1, datapoint2: value2, ...}
-> state
This structure is also easier to process for a watchdog.
Furthermore: let's say we have two topics like
hue1/status/lights/state -> 1.0
hue1/status/lights/hs -> {hue: 1.0, sat: 1.0}
floating around. When mode is switched to ct, the topic hue1/status/lights/hs becomes logically invalid (because the bulb doesn't have a hue anymore). Instead a new datapoint, containing the color temperature becomes relevant:
hue1/status/lights/ct -> 5000 (in Kelvin)
Here it would also be a better choice to publish all relevant datapoints in one topic, so that
hue1/status/lights -> { state: 1.0, hs: {hue: 1.0, sat: 1.0} }
would change to:
hue1/status/lights -> { state: 1.0, ct: 5000 }
This way other scripts need less logic than before. Formerly a subscriber needs to know, what to do when a ct message arrives (drop the old hs value out of its database). With ne new proposal we can just overwrite the whole state variable with the new topic payload, the contents of the topic will always contain the complete and most recent state.
We don't want to restrict developers into any scheme of topics. Therefore, a suggestion is to implement libraries that allow developers to choose the depth of their topic structure themselves. See test.js and test-result.png here.
However, we encourage to implement topic/datapoint combinations that make "sense", like mentioned above.
It is not neccessary to provide metadata for your topics, however it would be nice to have a way to provide documentation for other scripts about topic structure, units of values etc.
Getting back to the example above:
hue1/status/light -> { "hue":1.0, "sat":1.0, "bri":1.0 }
It's for every developer to decide with unit he wants to use. A hue (that doesn't have a unit) for e.g. can be expressed in many ways: 0 - 1.0, 0% - 100%, 0 - 65535 (Philips Hue), etc.
Developers are however encouraged to use physical units whenever possible (especially for sensors), e.g. temperature (°C), brightness (lux), power consumption (watts). For e.g. the brige for Homematic brightness sensors would need to implement a conversion.
It would also be nice to provide a description for contents of a topic. Homematic for e.g. provides a native description for "paramsets" (generated by hmGetInfo), like this:
"SET_TEMPERATURE": {
"VALUE": 4.5,
"CONTROL": "HEATING_CONTROL.SETPOINT",
"DEFAULT": 20,
"FLAGS": {
"VISIBLE": true,
"INTERNAL": false,
"TRANSFORM": false,
"SERVICE": false,
"STICKY": false
},
"ID": "SET_TEMPERATURE",
"MAX": 30.5,
"MIN": 4.5,
"OPERATIONS": {
"READ": true,
"WRITE": true,
"EVENT": true
},
"TAB_ORDER": 6,
"TYPE": "FLOAT",
"UNIT": "°C"
}
For an mqtt-smarthome, this description could look similar, e.g.:
{
"hue1/status/light": {
"hue": {
"default": 0,
"min": 0,
"max": 360,
"unit": "°",
"type": "int",
"refreshInterval": 10000
},
"sat": { //...
}
}
"topic": {
"datapoint1" : { ... },
"datapoint2" : { ... }
}
}
A single device can have multiple services. For e.g. the ESP8266 DevilRemote has an output that sets the speaker's volume and a DS18B20 sensor measuring the temperature. These two services could be:
devilremote123/_/hifi -> {"volume":1.0, "bass":"-1.0"}
devilremote123/_/temperature -> 21.0
The "1.0" topic structure is optimized for bridges:
bridge/_/deviceID/datapoint -> value
vs.
manufacturer/_/device_ID/service/datapoint -> value
bridge /_/device_ID/service/datapoint -> value
bridge/connected -> 0 | 1 | 2 (last will message)
vs.
manufacturer/maintenance/device_ID/available -> true | false (last will message)
bridgeName /maintenance/bridge /available -> true | false (last will message)
bridgeName /maintenance/device_ID/available -> true | false
Introduce maintenance status messages instead of the former connected:
device/maintenance/#
Topics below this level can be used to determine the health of a device or a bridge. For e.g. you could implement the topics
device/maintenance/available -> true | false (last will message, similar to the former /connected)
device/maintenance/uptime
device/maintenance/temperature/cpu
When a bridge service publishes retained topics of for e.g. a wirelessly connected device, the message will be preserved even if the device's battery dies or becomes unreachable in any other way.
Let's take a battery driven, wirelessly attached Homematic sensor as an example:
hm-MQ1234567-3/status/brightness -> 1000.0
It is known that the brightness is refreshed once every 6mins. Without retained messages, when a new client joins the MQTT broker, it won't know the current daylight brightness until the sensor broadcasts its state again. However, with retained messages this newly connected client will always get a value, even if the sensor becomes unavailable for any reason.
In MQTT-Smarthome "1.0" this was solved by introducing the ts datapoint. But this involves additional logic on the client side, because you have to check every message for its validity first. Let's say we have a watchdog, that either knows how long the interal of a sensor is from a database (mqtt-meta), calculates it by mesuring the average time inbetween messages or gets the value set via MQTT itself:
Bridge sends:
watchdog/set/timeout -> [ {"topic":"hm-MQ1234567-3/status/brightness", "seconds":360},
{"topic":"hm-MQ1234567-3/status/motion", "seconds":360} ]
The watchdog then sets a timeout for 6mins (plus tollerance) and subscribes to the topics. Whenever a new message arrives, the timout timer is reset, because the device is (obviously) still alive. When the message doesn't arrive anymore, it can mark the device as unavailable an post a null message to the according topic to clear the retained message, that now became invalid.
The watchdog can also subscribe to
+/maintenance/available
and clear the device's topics when a device marks itself as offline via last will.
The watchdog should run on the same host as the MQTT broker.