Conflicting bleak library version with HomeAssistant 2022.8

[carsonb]

It looks like 2022.8 has updated to bleak 0.15.0 home-assistant/core#75941

When trying to use 4.0.0 of sensor.airthings_wave I get the following error in the logs:

Unable to install package bleak==0.14.3: ERROR: Cannot install bleak==0.14.3 because these package versions have conflicting dependencies. ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

[ismarslomic]

Just upgraded HA to v2022.8.0 with already installed sensor.airthings_wave v4.0.0 and got same error statement in HA logs.

[lymanepp]

HA 2022.8 added native bluetooth support using the bleak library - https://www.home-assistant.io/blog/2022/08/03/release-20228/#first-class-bluetooth-support

I modified the manifest.json file to reference bleak 0.15.0, but it's still not working. Here's what the log shows.

2022-08-03 14:33:47.636 INFO (MainThread) [homeassistant.components.sensor] Setting up sensor.airthings_wave
2022-08-03 14:33:57.661 INFO (SyncWorker_5) [custom_components.airthings_wave.sensor] Found 1 airthings device(s)
2022-08-03 14:33:57.730 WARNING (MainThread) [homeassistant.components.sensor] Setup of sensor platform airthings_wave is taking over 10 seconds.
2022-08-03 14:33:57.749 INFO (SyncWorker_5) [custom_components.airthings_wave.airthings] Not able to connect to 04:EE:03:BB:B4:13/
2022-08-03 14:33:57.749 ERROR (SyncWorker_5) [custom_components.airthings_wave.airthings] Not getting device info because failed to connect to device.
2022-08-03 14:33:57.793 INFO (SyncWorker_5) [custom_components.airthings_wave.airthings] Not able to connect to 04:EE:03:BB:B4:13

[kongjudas]

Just a question: As the this integration is unsupported for the time being, will devs focus on the new BT integration, if even possible?

[lone-cloud]

bleak v0.15 is considered a potentially breaking release based on semserv versioning of a 0.x pre-release (aka unstable). I'm not a python dev, but it looks like poor debugging output as we don't actually know what the exception is here: https://github.com/custom-components/sensor.airthings_wave/blob/master/custom_components/airthings_wave/airthings.py#L245
I believe that anyone with the faintest knowledge of python should be able to easily solve this. If not, I'll work on this tomorrow, as I'd like to be on the latest release of HA with this integration working.

[lymanepp]

I added logging of the exception details:

This might be the root cause of the issue: got Future <Future pending> attached to a different loop

2022-08-03 21:44:38.041 INFO (MainThread) [homeassistant.components.sensor] Setting up sensor.airthings_wave
2022-08-03 21:44:48.062 INFO (SyncWorker_0) [custom_components.airthings_wave.sensor] Found 1 airthings device(s)
2022-08-03 21:44:48.157 INFO (SyncWorker_0) [custom_components.airthings_wave.airthings] Not able to connect to 04:EE:03:BB:B4:13
Traceback (most recent call last):
  File "/config/custom_components/airthings_wave/airthings.py", line 239, in connect
    ret = await self._dev.connect()
  File "/usr/local/lib/python3.10/site-packages/bleak/backends/bluezdbus/client.py", line 107, in connect
    device = await BleakScannerBlueZDBus.find_device_by_address(
  File "/usr/local/lib/python3.10/site-packages/bleak/backends/scanner.py", line 221, in find_device_by_address
    return await cls.find_device_by_filter(
  File "/usr/local/lib/python3.10/site-packages/bleak/backends/scanner.py", line 250, in find_device_by_filter
    async with cls(detection_callback=apply_filter, **kwargs):
  File "/usr/local/lib/python3.10/site-packages/bleak/backends/scanner.py", line 96, in __aenter__
    await self.start()
  File "/usr/local/lib/python3.10/site-packages/bleak/backends/bluezdbus/scanner.py", line 137, in start
    self._stop = await manager.active_scan(
  File "/usr/local/lib/python3.10/site-packages/bleak/backends/bluezdbus/manager.py", line 376, in active_scan
    reply = await self._bus.call(
  File "/usr/local/lib/python3.10/site-packages/dbus_next/aio/message_bus.py", line 305, in call
    await future
RuntimeError: Task <Task pending name='Task-3555' coro=<AirthingsWaveDetect.get_info() running at /config/custom_components/airthings_wave/airthings.py:259> cb=[_run_until_complete_cb() at /usr/local/lib/python3.10/asyncio/base_events.py:184]> got Future <Future pending> attached to a different loop
2022-08-03 21:44:48.170 ERROR (SyncWorker_0) [custom_components.airthings_wave.airthings] Not getting device info because failed to connect to device.

[lone-cloud]

I've been debugging on a fork (https://github.com/lone-cloud/sensor.airthings_wave) and I'm getting the exact same error. I would definitely agree that it's the root cause of the issue here. The eception implies that there are multiple threads running and python doesn't seem to like when variables set in one thread get re-assigned in another. I have not been able to figure it out yet.

[B1ob]

In my opinion we should adapt this component to the new bluetooth integration. However i'm quite busy at the moment and also lacking the experience here.

[MartyTremblay]

Tons of hints on how to update to bleak 0.15.0 here home-assistant/core@80a9659

[mjmccans]

While I have not done much digging yet, I wonder if this is potentially related to the issue addressed by flecmart's PR. I wonder that because I have tried the dev branches of my own related scripts (here and here) against Bleak 0.15.1 and they work without issue, but I did implement flecmart's changes to ensure that disconnect is always called. It may be worth looking into that angle.

[lone-cloud]

@mjmccans I don't understand how anything works for you because using bleak 0.15.1 will result in the same bleak version mismatch that we started out with. edit: I noticed that a new HA patch just came out (for me) which is on bleak 0.15.1. This custom component really needs to migrate to using the new bluetooth integration instead of calling bleak directly, as was suggested, otherwise we'll keep running into this issue.

flecmart's PR didn't solve the issue, but I noticed that it broke exception handling because he was raising exceptions that didn't originate from the base Exception class. You'll see a very unhelpful "exceptions must derive from BaseException" errors in the log without fixing the new raises.

I did look over home-assistant/core@80a9659 but I couldn't find anything relevant to the issue here. Saying that, I'm a (mostly) typescript dev and this is my first time playing with python and with HA custom integration dev in general. I'm now at my wit's end about this got Future <Future pending> attached to a different loop error and someone more familiar with python+asyncio should probably take a look at this point.

[mjmccans]

@lone-cloud I will start with a disclaimer that I am not expert in this area, so please keep that in mind when reviewing anything I say here. I did the initial conversion from Bluepy to Bleak, but I definitely made mistakes along the way (some of which may not have been found yet). In terms of my scripts, one is a script that is meant to run on a machine separate from Home Assistant and it would have its own version of Bleak. The use case for me was to have the script running on a Raspberry Pi closer to the Airthings device which would in turn send the sensor values to Home Assistant using MQTT. The other is that same script packaged as a Home Assistant add-on, which I made when I transitioned to Home Assistant OS. The add-on runs in its own container and also has its own version of Bleak, and Home Assistant exposes the DBUS socket to the container for communications with the bluetooth adapter. What I find is interesting is that these scripts appear to have no issue running with their own copies of Bleak 0.15.1. The reason I thought flecmart's PR could be in play was the remark about this looking like a potential threading issue, and that was also the only difference between my version of airthings.py and the one in this repository.

I wonder if the issue is that the new bluetooth integration is not playing nice with this custom integration and that is causing the issue. Ultimately @B1ob is likely correct that the right answer is to move to the new bluetooth integration, but it would be interesting to see if the issue goes away if you uninstall or disable the new Home Assistant bluetooth integration. That could help debug this issue, or show that I am completely out to lunch.

[lone-cloud]

I just re-tested on the latest HA with bleak 0.15.1 and the new bluetooth integration removed. It appears that the issue is indeed because of the new bluetooth integration. Unfortunately, when it's disabled this component still fails to connect due to Event loop is closed... new error message though, yay. I've been trying to create and assign a new event loop via asyncio, but to no avail. HA somehow has control of it and I haven't been able to figure out how to get it working...

[mjmccans]

Well, I guess that is progress. Probably obvious, but this link has some details and best practices for using the new bluetooth integration. It is not urgent for me to dig into at the moment because my scripts are still working, but I am interested in the topic and will try to dive in in the future. From the documentation it actually looks like it would be pretty easy to get auto-discovery working, which would be pretty useful.

[flecmart]

I was trying to find some documentation about how to use the new bluetooh integration as a developer. Do we even need a custom component after integrating the airthing devices with the new integration?

[B1ob]

Do we even need a custom component after integrating the airthing devices with the new integration?

Yes, as i understand it the integrations rely on the new bluetooth integration as a kind of middleware.

As stated on the bluetooth integration page under "Integrations that require exclusive use of the Bluetooth Adapter" this does not work in parallel with legacy integrations. I also thought it could be possible to introduce "async" to the sensor.py. However i still think adapting to the new integration is the best approach.

[vincegio]

I've played around a with this I think we need to wait for home-assistant/core#76342 to be merged as it adds polling possibilities to the new system.

Anyways, here's some of my progress 🎉

first time working on home assistant integrations 👶

[lymanepp]

API changes coming in 2022.9 - https://developers.home-assistant.io/blog/2022/08/08/bluetooth-api-changes/

[hubertron]

Like the others I have the same issue. Actually updated from Feb update.

[DavidS]

I downgraded back to 20.22.7 for now. Thank $deity for backups :-D

[blackest]

Back to 2022.7.7 for me also (HA 2022.8.3 is using bleak 0.15.1) HA 2022.8.3 breaks more than just this component and it probably doesnt help i'm using python 3.9x as well.

Funny coincidence bleak 0.15.1 development was sponsered by nabucasa.com

[BladeBear]

https://github.com/mjmccans/hassio-addon-airthings, this is working with Airthings wave with current Home Assistant release.

[MartyTremblay]

Thanks for the info @BladeBear,

However, the project link you provided is an add-on and not an HA component which means that it doesn't depend on the same bleak instance as the one built into HA.

None the less, this MQTT approach clearly works and could be a viable solution until we get our component working again.

[lone-cloud]

I've gotten hard stuck on the new way to connect to BLE devices:
BleakClient(bluetooth.async_ble_device_from_address(hass, mac))
The new docs say that the BleakClient must now be passed in a BLEDevice and the bluetooth.async_ble_device_from_address(hass, mac) resolves as expected, but when passing it into the BleakClient it just hangs forever for me with no debugable trace. Not sure if anyone's run into this issue. I searched through the hassio core integrations, but nowhere else do they try to use the BleakClient, other than the bluetooth integration.

[Jc2k]

HA integrations in core have to push some of their logic for talking to a device to a library on pypi. So what you'll probably find is stuff like this where the bledevice is passed to an external library and that is where the BleakClient lives.

There will also be logic in there to handle if the BLEDevice changes (via bluetooth.async_register_callback maybe). For example, in a future release you can have remote proxies (remote bluetooth dongles over usbip or EPSHome etc). AIUI, the BLEDevice might change as it self optimises the best proxy to use for each device.

What does the output from async_ble_device_from_address look like for you?

I don't know if it will help but make sure that your custom component depends on the bluetooth integration in its manifest.json. What does your logging config look like? And what bluetooth adapter are you using (we are having a haaard time with realtek atm).

[vincegio]

I started working on a draft to try and get this into core.
https://github.com/vincegio/airthings-ble

That's inspired from the Bluetooth devices repos, but since there are no advertisements, it's closer to PySwitchbot. Got some minor help from bdraco where to find the docs and how to think around this.

It works and the core component is kind of done. But haven't been able to commit it to my fork due to pre-commit errors for an unrelated file. Anyways, that's irrelevant for this issue though :)

[lymanepp]

I started working on a draft to try and get this into core.
https://github.com/vincegio/airthings-ble

Very clean!

[vincegio]

Thanks!
Here's the commit in core. Still got some stuff to do but it works vincegio/core@a07d07b

[B1ob]

Hi @vincegio nice work :) i had a first look at your integration. My feedback is the following:

• There is a bug in _get_service_characteristics which is missing a "return device" statement.
• I would add the address (mac) to the enity_id as with multiple devices the id would not be unique anymore (Or ask the user during config flow for a unique name).

[vincegio]

Hi @vincegio nice work :) i had a first look at your integration. My feedback is the following:

• There is a bug in _get_service_characteristics which is missing a "return device" statement.
• I would add the address (mac) to the enity_id as with multiple devices the id would not be unique anymore (Or ask the user during config flow for a unique name).

Thank you 👍 The entity part makes sense, thought it'd just add _2 at the end or something.

[tgmos]

Tried the instruction above:
ERROR (MainThread) [custom_components.airthings_ble.config_flow] Unknown error occurred from B0:91:xx:xx:xx:xx: 'date_time'

Logger: custom_components.airthings_ble.config_flow
Source: custom_components/airthings_ble/config_flow.py:92
Integration: airthings_ble
First occurred: 21:40:03 (2 occurrences)
Last logged: 21:40:25

[karepiu]

Was it added to 2022.9 ? Is there a HACS module ?

[nehalvpatel]

It looks like it didn't make it into 2022.9.

[Jc2k]

No it didn't. It was very close, but some of the HA developers wanted to wait until we had better UX for the case where there are 2 integrations for the same devices. We pushed for an approach that shouldn't cause @vincegio too much extra hassle, so I hope this is only a minor blip and we can get it in next month.

RE HACS - Personally, I would try and get code review done before publishing anything standalone, in case we do end up needing to make breaking changes to the unique identifiers or the config entry data. When those things go wrong it can make a bit of a mess (i've had to hand edit JSON to repair this sort of thing before). If people want to take that risk for themselves, they can grab the code from the branch. Its still fairly easy, but they will be more aware that if it breaks they get to keep the pieces.

[karepiu]

bummer. I decided to wait for it to get 2022.9 but now I am seriously considering grabbing the code. Thanks for the update

[Sticky12]

I am currently getting this error after updating to 2022.9

Logger: homeassistant.util.package
Source: util/package.py:98
First occurred: 23.05.03 (3 occurrences)
Last logged: 23.05.17

Unable to install package airthings-ble==0.5.0: ERROR: Cannot install airthings-ble==0.5.0 because these package versions have conflicting dependencies. ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

[ismarslomic]

Just updated HA to v2022.9.0 and experienced the same error in logs for the airthings_ble component (latest version from master).

@vincegio any idea on what this could be? I see that airthings-ble module is available from pypi.

022-09-08 13:26:11.503 ERROR (MainThread) [homeassistant.setup] Setup failed for custom integration airthings_ble: Requirements for airthings_ble not found: ['airthings-ble==0.5.0'].
homeassistant.requirements.RequirementsNotFound: Requirements for airthings_ble not found: ['airthings-ble==0.5.0']

[Sticky12]

It might be bleak again, new ha version seems to use 0.16

[Jc2k]

Sort of. airthings-ble follows the other HA bluetooth projects in using poetry instead of pip. And when you add a dependency to your project in comes out looking like this in pyproject.toml:

bleak = "^0.15.1"
bleak-retry-connector = "^1.8.0"

The ^ is signficant. If memory serves, this says airthings-ble works with 0.15.1, 0.15.2, 0.15.whatever, but NOT 0.16. Same applies to bleak-retry-connector.

@vincegio could swap the ^ for a >= which means "or any later version". Thats what I do in xiaomi_ble and homekit_controller.

[vincegio]

I'll try to do some updates asap. As well with getting the PR reviews completed.

I'll think about making a hacs component, but could also include some kind of warning about it.

[Sticky12]

I have temporarily solved it, by merging the airthings-ble files from https://github.com/vincegio/airthings-ble into the custom_components/airthings_ble folder and removed the dependency on airthings-ble in the manifest.json file.

[vincegio]

For those who have copied the component manually, head on over to the manifest and update:

  "requirements": ["airthings-ble==0.5.1"],

[ismarslomic]

Thanks @vincegio! Just of curiosity, why did you extract parser and constants in separate python module, in stead of keeping it together with the airthings_ble component in HA core?

[Jc2k]

It's a hard HA rule.

I had to do it for one of my integrations where I made like one HTTP get and decode JSON to a dict.

Although tedious in more extreme cases, the point is that integrations are meant to be about data mapping/translation, and your library is meant to do all the hard stuff talking to the device.

[ismarslomic]

Alright, that makes sense!

[jla1710]

Is there any progress with this?

I'm still getting same error in

Home Assistant 2022.9.6
Supervisor 2022.09.1
Operating System 9.0
Frontend 20220907.2 - latest

[Jc2k]

@vincegio's PR was merged. It looks like it just missed the beta cut off. I'd ask for a backport but there are a few more tweaks requested on it. So as it stands it'll be in 2022.11.

[kbahey]

I just updated HA to 2022.10.3, and I can't get HA to start with airthings_wave.
That is because HA comes with bleak==0.18.1

If I change airthings_wave/manifest.txt to make bleak that version, it reports an error:

Logger: homeassistant.helpers.frame
Source: helpers/frame.py:77
First occurred: 8:33:59 PM (1 occurrences)
Last logged: 8:33:59 PM

Detected integration that attempted to call BleakClient with an address instead of a BLEDevice. 
Please report issue to the custom integration author for airthings_wave using this method at
 custom_components/airthings_wave/airthings.py, line 238: self._dev = BleakClient(mac.lower())

I think it is an API change in bleak, and HA has the new call, while airthings_wave is using an old call.

What can one do now?

[DavidS]

@kbahey a "new"/ported airthings integration is on path of getting released as part of 2022.11. until that happens you need to roll back to a HA version that supports this component.

[kbahey]

Thanks for the reply.

In the meantime, for others who can't wait ... I found a standalone solution that works provided that you have MQTT running (which I do).

It is airthings-mqtt-ha.

Install paho-mqtt and bleak using pip3, run the above Python script once to generate a configuration file, edit the file to your liking, then run it again as a daemon.

Add MQTT sensors for each item that is returned by the AirThings Wave Plus, and you are done.

[jla1710]

@kbahey a "new"/ported airthings integration is on path of getting released as part of 2022.11. until that happens you need to roll back to a HA version that supports this component.

@DavidS
What is the estimated release date for 2022.11?

[Jc2k]

https://www.home-assistant.io/faq/release/

[ghgeiger]

The new built-in integration is working great in 2022.11!

Thanks for all the hard work!

[mjmccans]

I have moved from my addon to the new integration and it is working well. The auto-discovery is also fantastic from a user experience perspective. Thank you for all the work getting this into Core.

The only minor gripe that I have is rounding (or lack thereof) of the sensor values. Given the sensors in the device my personal preference from a significant digits perspective (and what my addon did) is to round all of the values to not include any decimals except for the temperature that I would round to one decimal point. I am not sure if anyone else agrees, but for now I have just added some template sensors that do the rounding in my configuration file.

[jla1710]

Thanks for new improved integration. However, I still have some issues...

The new Airthing BLE intergrations finds my Airthings wave plus immediately and starts showing values from the device. After a few minutes, Home Assistant stops showing any values and and connection is lost. "Entity is unavailable" placeholder remains until I reboot the server os reinstall the integration.

I checked from the terminal using bluetootctl scan that device is broadcasting information all the time. For some reason (unknown to me) it just does not work.

Log entry:

Unable to fetch data: [MAC address]. Failed to connect, device was not found, It may have been removed from BlueZ when scanning stopped.

Error fetching airthings_ble data: Unable to fetch data: 'NoneType' object has no attribute 'address'

Any ideas whats going wrong here?

[Jc2k]

The right place to get help with integrations in HA Core is the HA GitHub. But that error suggests to me that it's actually outside the range of your bluetooth. Range in bluetooth is asymmetric - you can have enough signal to see some advertisements, but it be too weak for 2-way communication, which makes using bluetoothctl to debug this challenging. What sort of RSSI can you see in the bluetoothctl output you mention?

[jla1710]

@Jc2k

I need to investigate more thoroughly when I get home and check the RSSI levels. The direct distance is very short between RPI and Airthings wave but they are located in different floors. I have ESP32s scattered around the building also, maybe I can use one of them to communicate with AirThings?