pydeconz BridgeBusy 901 Internal error 951, all Zigbee devices are unresponsive

[GitHubxtsnl]

The problem

After approx. 48h after a full reboot all my (86) Zigbee devices are unresponsive. Only a full reboot of my host solves the issue temporarily. Reloading the Phoscon integration does not solve the issue.

running HA CORE 2024.5.4
Supervisor 2024.05.1
OS 12.3 (on a Pi4)
running deCONZ 6.23.0
running ConBeeII 2.25.3 / 26720700

What version of Home Assistant Core has the issue?

2024.5.4

What was the last working version of Home Assistant Core?

2024.4.x

What type of installation are you running?

Home Assistant Core

Integration causing the issue

deCONZ/Phoscon

Link to integration documentation on our website

https://www.home-assistant.io/integrations/deconz

Diagnostics information

config_entry-deconz-34acecef273c489e96809a07bfbfc6f4.json

Example YAML snippet

No response

Anything in the logs that might be useful for us?

Log Details (ERROR)
Logger: homeassistant.components.websocket_api.http.connection
Source: components/websocket_api/commands.py:239
integration: Home Assistant WebSocket API(documentation, issues) 
First occurred: 01:36:45 (17 occurrences) 
Last logged: 06:05:05
•	[2711362472] Unexpected exception
•	[2282634408] Unexpected exception
•	[2062853160] Unexpected exception
•	[2163144232] Unexpected exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/pydeconz/gateway.py", line 141, in request_with_retry
    return await self.request(method, path, json)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pydeconz/gateway.py", line 166, in request
    response: dict[str, Any] = await self._request(
                               ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pydeconz/gateway.py", line 192, in _request
    _raise_on_error(response)
  File "/usr/local/lib/python3.12/site-packages/pydeconz/gateway.py", line 223, in _raise_on_error
    raise_error(data["error"])
  File "/usr/local/lib/python3.12/site-packages/pydeconz/errors.py", line 73, in raise_error
    raise cls(
pydeconz.errors.BridgeBusy: 901 /lights/8/state/on Internal error, 951

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/pydeconz/gateway.py", line 141, in request_with_retry
    return await self.request(method, path, json)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pydeconz/gateway.py", line 166, in request
    response: dict[str, Any] = await self._request(
                               ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pydeconz/gateway.py", line 192, in _request
    _raise_on_error(response)
  File "/usr/local/lib/python3.12/site-packages/pydeconz/gateway.py", line 223, in _raise_on_error
    raise_error(data["error"])
  File "/usr/local/lib/python3.12/site-packages/pydeconz/errors.py", line 73, in raise_error
    raise cls(
pydeconz.errors.BridgeBusy: 901 /lights/8/state/on Internal error, 951

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/pydeconz/gateway.py", line 141, in request_with_retry
    return await self.request(method, path, json)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pydeconz/gateway.py", line 166, in request
    response: dict[str, Any] = await self._request(
                               ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pydeconz/gateway.py", line 192, in _request
    _raise_on_error(response)
  File "/usr/local/lib/python3.12/site-packages/pydeconz/gateway.py", line 223, in _raise_on_error
    raise_error(data["error"])
  File "/usr/local/lib/python3.12/site-packages/pydeconz/errors.py", line 73, in raise_error
    raise cls(
pydeconz.errors.BridgeBusy: 901 /lights/8/state/on Internal error, 951

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 239, in handle_call_service
    response = await hass.services.async_call(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2738, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2779, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 975, in entity_service_call
    single_response = await _handle_entity_call(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1047, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 627, in async_handle_light_on_service
    await light.async_turn_on(**filter_turn_on_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/deconz/light.py", line 301, in async_turn_on
    await self.api.set_state(id=self._device.resource_id, **data)
  File "/usr/local/lib/python3.12/site-packages/pydeconz/interfaces/lights.py", line 172, in set_state
    return await self.gateway.request_with_retry(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pydeconz/gateway.py", line 154, in request_with_retry
    return await self.request_with_retry(method, path, json, tries)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pydeconz/gateway.py", line 154, in request_with_retry
    return await self.request_with_retry(method, path, json, tries)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/pydeconz/gateway.py", line 157, in request_with_retry
    raise BridgeBusy
pydeconz.errors.BridgeBusy

Additional information

I'm almost certain this error occured when I upgraded from 2024.4.x to 2024.5 release.

[home-assistant]

Hey there @Kane610, mind taking a look at this issue as it has been labeled with an integration (deconz) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of deconz can trigger bot actions by commenting:

• @home-assistant close Closes the issue.
• @home-assistant rename Awesome new title Renames the issue.
• @home-assistant reopen Reopen the issue.
• @home-assistant unassign deconz Removes the current integration label and assignees on the issue, add the integration domain after the command.
• @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue.
• @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

---

deconz documentation
deconz source
_{^{(message by IssueLinks)}}

[Kane610]

The error comes from deconz and says bridge busy, so its probably queueing up calls to your light with id 8, so the issue is probably with the light or deconz