Expose device configuration after hass is online

[urpylka]

Hi, I solved the problem with device resending device configuration after restarting Homeassistant

Short story long: It is about Zigbee devices unavailable (in Homeassistant 2023.12.3) after Home Assistant restarting.

I added calling the discover(entity, forced) when we get online message in homeassistant state topic.

I am not sure about using next section:

if (entity.isDevice() && this.discoveredTriggers[entity.ieeeAddr]) {
    for (const config of this.discoveredTriggers[entity.ieeeAddr]) {
        const key = config.substring(0, config.indexOf('_'));
        const value = config.substring(config.indexOf('_') + 1);
        this.publishDeviceTriggerDiscover(entity, key, value, true);
    }
}

Please let me know if some changes I should to add to the PR or edit my PR by yourself (I allow it).

Source: #18862

[urpylka]

As I see we need to cover by tests on homeassistant.ts 1688-1691.

But the problem that I have never wrote on TypeScript and covered smth by tests, as I see we need smth like that https://github.com/Koenkk/zigbee2mqtt/blob/master/test/homeassistant.test.js#L78 or just delete the section 1687-1694.

Please assist

[Koenkk]

With this PR, all discovery messages will be send again everytime HA comes online. We already have issues with this: #20648

Short story long: It is about Zigbee devices unavailable (in Homeassistant 2023.12.3) after Home Assistant restarting.

I also don't understand how this fixes this problem. MQTT messages should be retained so Z2M shouldn't have to send them again. I would expect that just resending the device availability would also fix this issue?

[urpylka]

Hi, let me investigate it this week. Will return with answer.

[urpylka]

The basis for understanding how Homeassistant works with mqtt is this instruction.

After configuring MQTT integration in Homeassistant besides the ability to use topics in manually described devices, there is the ability to automatically discover and configure new devices.

The most important information related to this case is described in How to use discovery messages. I will make some quotes from it:

When Home Assistant is restarting, discovered MQTT items with a unique ID will be unavailable until a discovery message is received.

There are two approaches to make sure the discovered items are set up at startup:

Using Birth and Will messages to trigger setup
Using retained messages

You were right – one way is using retained messages.

After that, I tested subscriptions of Homeassistant after restarting at webui of my broker:

As you see it has:

homeassistant.*.*.*.config
homeassistant.*.*.config

It means that any messages that devices are available will not be consumed by Homeassistant - there are no subscriptions to these topics (for ex zigbee2mqtt/0xa4c138bae2333cf1).

I don't know how next problems relate to another brokers, but I use RabbitMQ 3.8.19 and it has next ones:

• No support to send retained messages by a mask.
• The broker will flush retained messages after restarting.
• Finally there is a problem with working retained messages on HA setup it will only work on the same node.

Sources:

• Retained messages are only delivered to clients for which a queue currently exists
• MQTT: replicated message retention

After this investigation my opinion that we need to implement resending discover information when get Birth message (and probably can delete retain flag). Or second way - disable discovering entities by mqtt in Homeassistant and configure it manual.

[Koenkk]

@urpylka this sounds like a broker issue? Can you try with mosquitto?

[mundschenk-at]

Both Mosquitto and EMQX can persist retained messages over a broker restart.

[urpylka]

@Koenkk I tested as you asked on Mosquitto regarding subscription by mask to get a retained message, and it works.

> mosquitto_sub -t '#' -v
test/status online

But I am not sure that it will change smth. Because Mosquitto doesn't pretend to be a production solution (it doesn't have High Availability functions). And I still use RabbitMQ with the known issue that I can't get retained message by mask.

If you see reason to use current approach to discover devices, maybe we need to create a configuration parameter to allow user to choose which trigger is use for discover (by retain mes / by birth mes)?

[corporategoth]

The problem seems to be that the ONLY topic being persisted is the bridge state.
To wit:

 mosquitto_sub -u test -P '****' -t 'homeassistant.*.*.*.config' -t 'homeassistant.*.*.config' -t 'zigbee2mqtt/bridge/stat
e' -t 'zigbee2mqtt/Office Closet Light'
{"state":"online"}

I get no data, no results. Home Assistant and Z2M ((and mosquitto) are all up.
This means, if I restart HA right now, it would not autoconfig (because that is triggered by Z2M's restarting, not HA's) because all the config topics are not persisted.
It would also get the current status of any device until they happened to update in Z2M (ie. the device itself sends an update).

This means

1. HA might not have the correct list of devices, as it would just be configured with whatever it was previously (and Z2M config could have changed while HA was offline).
2. HA would not have the current state of devices it DOES know about, because those device states are not set to be persisted in MQTT.
3. It should know things are online, but that's about it.

If I leave the above command running, I get updates on my office closet light state when it sends updates. But NOT at HA startup. And if that was a new device added while HA was down, HA would never know about it. Or if that device was REMOVED while HA was disconnected, HA would always remember it / think it still exists.

A number of versions ago (I don't remember which), HA used to send some kind of message to Z2M that triggered autoconfig again. But that doesn't appear to happen anymore. Either because HA stopped sending it, or Z2M stopped handling it. So right now, if HA restarts, I DO have to restart Z2M too to ensure autoconfig happens again.

[mundschenk-at]

The problem seems to be that the ONLY topic being persisted is the bridge state. To wit:

 mosquitto_sub -u test -P '****' -t 'homeassistant.*.*.*.config' -t 'homeassistant.*.*.config' -t 'zigbee2mqtt/bridge/stat
e' -t 'zigbee2mqtt/Office Closet Light'
{"state":"online"}

I get no data, no results.

That is not valid MQTT wildcard syntax, so it's no surprise you don't get any results.

[Koenkk]

@corporategoth is anything published to homeassistant/status when HA starts?

[mundschenk-at]

@corporategoth try this:

 mosquitto_sub -u test -P '****' -t 'homeassistant/+/+/+/config' -t 'homeassistant/+/+/config' -t 'zigbee2mqtt/bridge/state' -t 'zigbee2mqtt/Office Closet Light'

[corporategoth]

@corporategoth try this:

 mosquitto_sub -u test -P '****' -t 'homeassistant/+/+/+/config' -t 'homeassistant/+/+/config' -t 'zigbee2mqtt/bridge/state' -t 'zigbee2mqtt/Office Closet Light'

Aha, that works. Then I get >10k entries.

@Koenkk
Yeah, when I just restarted my HA, I got

offline
online

So apparently it does publish there. Not a JSON blob, just the words themselves.

HOWEVER, if I look at most of the config options on the device in question above, MOST of them are unavailable in HA.
Zeroing in on 1 control, 'ActiveEnergyReports', I noticed I have 2 config messages returned by that wildcard above:

homeassistant/sensor/0x6c5cb1fffe5e7187/activeEnergyReports/config:
{"availability":[{"topic":"zigbee2mqtt/bridge/state","value_template":"{{ value_json.state }}"}],"device":{"identifiers":["zigbee2mqtt_0x6c5cb1fffe5e7187"],"manufacturer":"Inovelli","model":"Inovelli 2-in-1 switch + dimmer (VZM31-SN)","name":"Office Closet Light","sw_version":"2.15","via_device":"zigbee2mqtt_bridge_0xe0798dfffea88b0a"},"enabled_by_default":false,"name":"ActiveEnergyReports","object_id":"office_closet_light_activeEnergyReports","origin":{"name":"Zigbee2MQTT","sw":"1.35.3","url":"https://www.zigbee2mqtt.io"},"state_topic":"zigbee2mqtt/Office Closet Light","unique_id":"0x6c5cb1fffe5e7187_activeEnergyReports_zigbee2mqtt","value_template":"{{ value_json.activeEnergyReports }}"}

homeassistant/number/0x6c5cb1fffe5e7187/activeEnergyReports/config:
{"availability":[{"topic":"zigbee2mqtt/bridge/state","value_template":"{{ value_json.state }}"}],"command_topic":"zigbee2mqtt/Office Closet Light/set/activeEnergyReports","device":{"identifiers":["zigbee2mqtt_0x6c5cb1fffe5e7187"],"manufacturer":"Inovelli","model":"Inovelli 2-in-1 switch + dimmer (VZM31-SN)","name":"Office Closet Light","sw_version":"2.15","via_device":"zigbee2mqtt_bridge_0xe0798dfffea88b0a"},"max":32767,"min":0,"name":"ActiveEnergyReports","object_id":"office_closet_light_activeEnergyReports","origin":{"name":"Zigbee2MQTT","sw":"1.35.3","url":"https://www.zigbee2mqtt.io"},"state_topic":"zigbee2mqtt/Office Closet Light","unique_id":"0x6c5cb1fffe5e7187_activeEnergyReports_zigbee2mqtt","value_template":"{{ value_json.activeEnergyReports }}"}

I wonder if, in my case, something is hanging around, the topic not cleaned up by Z2M. And one is overriding the other?

Meaning:

1. I would have to clear out my MQTT state entirely and re-build it to fix this, and
2. The logic Z2M has to clean up topics that are not/no longer valid is not working

This would also be why, if I restart Z2M, it will re-send config stuff for the correct topic only, which will kick HA to use the correct info.

Not sure what's going on in my case

[urpylka]

@corporategoth it is not a wildcard syntax (homeassistant.*.*.config). It is a routing key of RabbitMQ, but we can assume which wildcard it is suppose to be (as @mundschenk-at has wrote).

[Koenkk]

I expect the only thing needed to fix this is adding this.eventBus.emitPublishAvailability(); to

zigbee2mqtt/lib/extension/homeassistant.ts

Line 1660 in f04ade6

[urpylka]

I expect the only thing needed to fix this is adding this.eventBus.emitPublishAvailability(); to

zigbee2mqtt/lib/extension/homeassistant.ts

Line 1660 in f04ade6

I will check next week and let you know help it or not.

[popy2k14]

@urpylka any news on this?
Does the provided fix from Koenkk #21500 (comment) work?

Thanks a lot for investigating the issue.

[urpylka]

Hi, I am sorry for the delay, I've just tested this.eventBus.emitPublishAvailability(); and it doesn't work: after restarting devices are unavailable.

@Koenkk I understand that can consider it MQTT issue, but I think that we should implement another discovery mechanism by birth message from Homeassistant. (If this problem is so widespread). That approach doesn't have any problems for that from my perspective.

[xekil]

Good morning,
Is a fix coming?
THANKS

[Koenkk]

@urpylka

Hi, I am sorry for the delay, I've just tested this.eventBus.emitPublishAvailability(); and it doesn't work: after restarting devices are unavailable.

After adding this, do you see availability info being published to Z2M when HA starts?

[urpylka]

@Koenkk Hi, I am not sure what do you mean by availability info so after restart HA I got next strings in logs of Z2M:

Zigbee2MQTT:info  2024-04-10 09:35:10: MQTT publish: topic 'zigbee2mqtt/<some_dev_id>', payload '{"linkquality":111,"power_on_behavior":"previous","state":"OFF","switch_type":"toggle"}'
Zigbee2MQTT:info  2024-04-10 09:35:10: MQTT publish: topic 'zigbee2mqtt/bridge/state', payload '{"state":"online"}'

But for correct working Homeassistant we need to resend to it devices configs to topics: homeassistant/<dev_type>/<dev_id>/<type>/config

[Koenkk]

You should also see, e.g. zigbee2mqtt/bulb_color/availability with payload online. If that' is not the case, the this.eventBus.emitPublishAvailability(); was not used correctly.

[urpylka]

You should also see, e.g. zigbee2mqtt/bulb_color/availability with payload online. If that' is not the case, the this.eventBus.emitPublishAvailability(); was not used correctly.

Everything that I got after restart HA, I texted above

[Koenkk]

• Can you add, e.g. a console.log('POINT 1') just below

  zigbee2mqtt/lib/extension/availability.ts

  Line 133 in ffc2ff1

  @bind private publishAvailabilityForAllEntities(): void {

  to see if this function is called when adding this.eventBus.emitPublishAvailability();?
• With this.eventBus.emitPublishAvailability(); added, could you provide the z2m debug log when HA comes online?

See this on how to enable debug logging.