Pi4 RA Docker Bluetooth Crash

[mwolter805]

Thank you for your time developing this great software! Your contribution to the community has been immense and wanted to provide detailed issue info and resolution steps regarding this issue as it may affect many users.

PS Looking forward to your implementation to triangulate device and room location as well as an apple watch app for BLE!

Describe the bug
When using a Pi4 with PiOS version 2021-03-04-raspios-buster-armhf-lite with Home Assistant Docker version 2.15.1, BLE detection often crashes within 24 hours. Restarting the container or rebooting the Pi restores normal operation.

To reproduce
Let the container remain active longer than 24 hours.

Relevant logs
Docker logs of the container. These logs were pulled at 4/21/2021, 07:33:47 AM. As you can see there were no updates past 10:12 PM the previous night.

4/20/2021, 10:11:16 PM - info - BluetoothLowEnergyService: Discovered new BLE peripheral e5a392d853514fb48bde277773263d48-1337-38525 with ID e5a392d853514fb48bde277773263d48-1337-38525 and RSSI -91
4/20/2021, 10:12:11 PM - info - BluetoothLowEnergyService: Attempting app discovery for tag 5966b1b6f79c
4/20/2021, 10:12:15 PM - info - BluetoothLowEnergyService: Discovered new BLE peripheral 5966b1b6f79c with ID 5966b1b6f79c and RSSI -91
4/20/2021, 10:12:56 PM - warn - BluetoothService: Did not detect any low energy advertisements in a while, resetting

dmesg from the host Pi

[86446.521711] Bluetooth: hci0: advertising data len corrected 21 -> 6
[86452.714273] Bluetooth: hci0: link tx timeout
[86452.715877] Bluetooth: hci0: link tx timeout
[86454.575923] 8<--- cut here ---
[86454.579003] Unable to handle kernel NULL pointer dereference at virtual address 00000012
[86454.587401] pgd = 1ab4af04
[86454.590107] [00000012] *pgd=80000000004003, *pmd=00000000
[86454.595556] Internal error: Oops: 206 [#1] SMP ARM
[86454.600345] Modules linked in: veth xt_nat xt_tcpudp xt_conntrack xt_MASQUERADE nf_conntrack_netlink nft_counter xt_addrtype nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink br_netfilter bridge overlay cmac bnep sha256_generic cfg80211 8021q garp stp llc rtl2832_sdr r820t rtl2832 i2c_mux regmap_i2c dvb_usb_rtl28xxu dvb_usb_v2 dvb_core ftdi_sio cdc_acm usbserial sg bcm2835_codec(C) bcm2835_isp(C) v4l2_mem2mem v3d gpu_sched bcm2835_v4l2(C) videobuf2_dma_contig bcm2835_mmal_vchiq(C) videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 vc4 videobuf2_common cec drm_kms_helper rpivid_mem raspberrypi_hwmon videodev mc btusb btrtl vc_sm_cma(C) btintel btbcm drm drm_panel_orientation_quirks bluetooth snd_soc_core ecdh_generic snd_compress snd_pcm_dmaengine ecc rfkill snd_pcm snd_timer snd syscopyarea sysfillrect sysimgblt fb_sys_fops uio_pdrv_genirq backlight uio nvmem_rmem ip_tables x_tables ipv6
[86454.683088] CPU: 0 PID: 29302 Comm: kworker/u9:2 Tainted: G         C        5.10.11-v7l+ #1399
[86454.691784] Hardware name: BCM2711
[86454.695227] Workqueue: hci0 hci_rx_work [bluetooth]
[86454.700141] PC is at hci_connect_le_scan_cleanup+0x14/0x12c [bluetooth]
[86454.706786] LR is at create_le_conn_complete+0xdc/0xe0 [bluetooth]
[86454.712962] pc : [<bf1f4794>]    lr : [<bf1f6df0>]    psr: 60070013
[86454.719225] sp : c8935dc0  ip : c8935de0  fp : c8935ddc
[86454.724444] r10: c31f3780  r9 : c2cc2008  r8 : c1205048
[86454.729664] r7 : c2cc2008  r6 : 00000000  r5 : c2cc2000  r4 : 00000000
[86454.736187] r3 : c2cc2920  r2 : 00200400  r1 : 00000000  r0 : 00000000
[86454.742711] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[86454.749842] Control: 30c5383d  Table: 05c2c180  DAC: fffffffd
[86454.755584] Process kworker/u9:2 (pid: 29302, stack limit = 0x91b4dc11)
[86454.762195] Stack: (0xc8935dc0 to 0xc8936000)
[86454.766548] 5dc0: 00000000 c2cc2000 00000000 c2cc2008 c8935dfc c8935de0 bf1f6df0 bf1f478c
[86454.774724] 5de0: c2cc2000 c31f3c00 0000000e c6d9580a c8935eb4 c8935e00 bf1fd5a4 bf1f6d20
[86454.782900] 5e00: c8935e4c c8935e50 c8a2b0ac c76a2180 c31f3180 c1bbe200 c8935e3c c8935e28
[86454.791076] 5e20: c026f958 c026f7b0 000000c3 c09d5864 c8935e54 c8935e40 c09d5864 00000000
[86454.799251] 5e40: c76a2180 00cc2000 c31f2043 bf1f6d14 00000000 c8935e60 c09de360 c09df060
[86454.807427] 5e60: c76a2180 c09de3cc c8935e8c c8935e78 c09de3cc c09dcf40 00000000 c2cc2000
[86454.815602] 5e80: c8935eb4 3d1041cf bf20ff40 c2cc272c c2cc272c c2cc2000 c2cc2804 c31f3c00
[86454.823778] 5ea0: 00000000 c1bbe200 c8935ef4 c8935eb8 bf1f0210 bf1fd20c 00000000 bf23661c
[86454.831953] 5ec0: bf236650 c2cc200c c032c948 c2cc272c c89e2080 c1bbe200 00000120 c26c0100
[86454.840129] 5ee0: 00000000 c1bbe200 c8935f34 c8935ef8 c023da24 bf1f0154 c032cbf4 c020caa8
[86454.848304] 5f00: c1bbe200 0000000c c1bbe200 c89e2080 c89e2094 c1bbe200 00000088 c1bbe218
[86454.856479] 5f20: c1203d00 c1bbe200 c8935f74 c8935f38 c023ddb4 c023d7dc c0b6918c c0e1c3ec
[86454.864655] 5f40: c133a92e ffffe000 c8935f74 c8a12880 c74094c0 00000000 c8934000 c023dd54
[86454.872830] 5f60: c89e2080 c89e5e74 c8935fac c8935f78 c024583c c023dd60 c8a128a4 c8a128a4
[86454.881006] 5f80: ffffe000 c74094c0 c02456cc 00000000 00000000 00000000 00000000 00000000
[86454.889181] 5fa0: 00000000 c8935fb0 c02000ec c02456d8 00000000 00000000 00000000 00000000
[86454.897356] 5fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[86454.905532] 5fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[86454.913706] Backtrace: 
[86454.916221] [<bf1f4780>] (hci_connect_le_scan_cleanup [bluetooth]) from [<bf1f6df0>] (create_le_conn_complete+0xdc/0xe0 [bluetooth])
[86454.928132]  r7:c2cc2008 r6:00000000 r5:c2cc2000 r4:00000000
[86454.933854] [<bf1f6d14>] (create_le_conn_complete [bluetooth]) from [<bf1fd5a4>] (hci_event_packet+0x3a4/0x2cfc [bluetooth])
[86454.945069]  r7:c6d9580a r6:0000000e r5:c31f3c00 r4:c2cc2000
[86454.950790] [<bf1fd200>] (hci_event_packet [bluetooth]) from [<bf1f0210>] (hci_rx_work+0xc8/0x2a8 [bluetooth])
[86454.960790]  r10:c1bbe200 r9:00000000 r8:c31f3c00 r7:c2cc2804 r6:c2cc2000 r5:c2cc272c
[86454.968617]  r4:c2cc272c
[86454.971184] [<bf1f0148>] (hci_rx_work [bluetooth]) from [<c023da24>] (process_one_work+0x254/0x584)
[86454.980229]  r10:c1bbe200 r9:00000000 r8:c26c0100 r7:00000120 r6:c1bbe200 r5:c89e2080
[86454.988055]  r4:c2cc272c
[86454.990586] [<c023d7d0>] (process_one_work) from [<c023ddb4>] (worker_thread+0x60/0x5c4)
[86454.998675]  r10:c1bbe200 r9:c1203d00 r8:c1bbe218 r7:00000088 r6:c1bbe200 r5:c89e2094
[86455.006501]  r4:c89e2080
[86455.009032] [<c023dd54>] (worker_thread) from [<c024583c>] (kthread+0x170/0x174)
[86455.016426]  r10:c89e5e74 r9:c89e2080 r8:c023dd54 r7:c8934000 r6:00000000 r5:c74094c0
[86455.024253]  r4:c8a12880
[86455.026783] [<c02456cc>] (kthread) from [<c02000ec>] (ret_from_fork+0x14/0x28)
[86455.034002] Exception stack(0xc8935fb0 to 0xc8935ff8)
[86455.039049] 5fa0:                                     00000000 00000000 00000000 00000000
[86455.047224] 5fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[86455.055399] 5fe0: 00000000 00000000 00000000 00000000 00000013 00000000
[86455.062010]  r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c02456cc
[86455.069837]  r4:c74094c0
[86455.072366] Code: e92dd8f0 e24cb004 e52de004 e8bd4000 (e5d06012) 
[86455.078644] ---[ end trace d8fc6ef350cd8391 ]---
[86497.631252] Bluetooth: hci0: HCI reset during shutdown failed

cat /proc/version

Linux version 5.10.11-v7l+ (dom@buildbot) (arm-linux-gnueabihf-gcc-8 (Ubuntu/Linaro 8.4.0-3ubuntu1) 8.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #1399 SMP Thu Jan 28 12:09:48 GMT 2021

Relevant configuration
Docker run command

sudo docker run -d --name=room-assistant \
--net=host \
--restart=always \
--cap-add=NET_ADMIN \
--cap-add=SYS_ADMIN \
-e TZ=America/Los_Angeles \
-v $HOME/room-assistant:/room-assistant/config \
mkerix/room-assistant:2.15.1

Room Assistant Config

global:
  instanceName: tablet01
  integrations:
    - mqtt
    - bluetoothLowEnergy
cluster:
  autoDiscovery: false
  weight: 10
  port: 6425
  peerAddresses:
    - 192.168.8.71:6425
    - 192.168.8.72:6425
    - 192.168.8.73:6425
    - 192.168.8.10:6425
bluetoothLowEnergy:
  hciDeviceId: 0
  timeout: 60
  allowlist:
    - 'DE2BE950-F59A-49FD-A582-42B56C29B32C'
  tagOverrides:
    'DE2BE950-F59A-49FD-A582-42B56C29B32C':
      name: Matts iPhone
mqtt:
  mqttUrl: mqtt://192.168.8.12:1883
  mqttOptions:
    username: ***
    password: ***
  retain: false

Expected behavior
Not to crash 🙂

Environment

• room-assistant version: 2.15.1
• installation type: Docker
• hardware: Raspberry Pi 4
• OS: Pi OS
• Using an external BT adapter version 10 manufacturer 93 class 0x2c0000

Additional context
Found issue #314 (comment) with comment by @gmalbert that appears related.

Pi appeared to be running an affected version.

# sudo apt list --installed | grep bluez-firmware
    bluez-firmware/testing,now 1.2-4+rpt8 all [installed,automatic]

Ran the following commands on the Pi4.

wget http://archive.raspberrypi.org/debian/pool/main/b/bluez-firmware/bluez-firmware_1.2-4+rpt2_all.deb
sudo dpkg -i bluez-firmware_1.2-4+rpt2_all.deb

I will add updates to this issue regarding the status of this fix.

[gmalbert]

@mwolter805 I would also add that you shouldn't upgrade the package when you do an upgrade of other components (or you'll be in the same spot again). The following command worked for me:

echo "bluez-firmware hold" | sudo dpkg --set-selections

Curious to see if this resolves things for you.

[mwolter805]

@gmalbert Thank you, yes, I will do that.

So far the Pi4 has been up for 23 hours continuously. The longest stint since installation. 👍

[mKeRix]

Thanks for posting the in-depth information about the issue and a possible fix! I'll pin this issue for more visibility.

[ghzgod]

This is also happening on all my PiZero-Ws suddenly. Finally got some time to troubleshoot and I noticed bluetooth classic stopped picking up devices completely. Just FYI for everyone

[corgan2222]

This is also happening on all my PiZero-Ws suddenly. Finally got some time to troubleshoot and I noticed bluetooth classic stopped picking up devices completely. Just FYI for everyone

did you tryed the fix with the older bluez-firmware ?
i have the same problem on several devices.

[sotatech]

The title on this issue is a bit misleading. In my experience, it applies to model 3 and zero's as well. Reverting to the older bluez-firmware fixed the problem on all my devices.
Thanks!