segmentation fault while serializing with 3.10.1 on x86_64/aarch64 with cpython 3.12

[bdraco]

• previous version 3.9.15 did not appear to have the issue
• no reproducer available yet (sorry)
• only seems to happen during serialize
• orjson.Fragment is being used, but that may not be relevant
• appears to only happen on x86_64 and aarch64
• crash is rare on some systems, and nearly immediate on others
• Home Assistant issue Upgrading to 2024.5.0b0 fails and auto rollback to 2024.4.4 or boot-loop home-assistant/core#116131 (comment)

[bdraco]

Summary of user reported Fatal Python error: Segmentation fault

Note: I cannot reproduce this locally so I do not have a reproducer or trace available yet

https://github.com/home-assistant/core/blob/0c44051d2ab8a78d8b979aaf950f5dc1c5c5bfa4/homeassistant/helpers/json.py#L158

Current thread 0x00007fe9aa0a0b38 (most recent call first):
  File "/usr/src/homeassistant/homeassistant/helpers/json.py", line 158 in _orjson_bytes_default_encoder
  File "/usr/src/homeassistant/homeassistant/helpers/json.py", line 188 in save_json
  File "/usr/src/homeassistant/homeassistant/helpers/storage.py", line 544 in _write_data
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58 in run
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 92 in _worker
  File "/usr/local/lib/python3.12/threading.py", line 1010 in run
  File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

https://github.com/home-assistant/core/blob/0c44051d2ab8a78d8b979aaf950f5dc1c5c5bfa4/homeassistant/helpers/json.py#L158

Current thread 0x00007f66a3f85b30 (most recent call first):
  File "/usr/src/homeassistant/homeassistant/helpers/json.py", line 158 in _orjson_bytes_default_encoder
  File "/usr/src/homeassistant/homeassistant/helpers/json.py", line 188 in save_json
  File "/usr/src/homeassistant/homeassistant/helpers/storage.py", line 544 in _write_data
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58 in run
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 92 in _worker
  File "/usr/local/lib/python3.12/threading.py", line 1010 in run
  File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

https://github.com/home-assistant/core/blob/0c44051d2ab8a78d8b979aaf950f5dc1c5c5bfa4/homeassistant/helpers/json.py#L158

Current thread 0x00007f74562ffb30 (most recent call first):
  File "/usr/src/homeassistant/homeassistant/helpers/json.py", line 158 in _orjson_bytes_default_encoder
  File "/usr/src/homeassistant/homeassistant/helpers/json.py", line 188 in save_json
  File "/usr/src/homeassistant/homeassistant/helpers/storage.py", line 544 in _write_data
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58 in run
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 92 in _worker
  File "/usr/local/lib/python3.12/threading.py", line 1010 in run
  File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

https://github.com/home-assistant/core/blob/0c44051d2ab8a78d8b979aaf950f5dc1c5c5bfa4/homeassistant/helpers/entity_registry.py#L324
https://github.com/home-assistant/core/blob/6bff0c384ffb1d26d21b52b19eb92439549beb28/homeassistant/helpers/json.py#L70

Current thread 0x00007fd26097fb30 (most recent call first):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_registry.py", line 325 in as_storage_fragment
  File "/usr/local/lib/python3.12/functools.py", line 995 in __get__
  File "/usr/src/homeassistant/homeassistant/helpers/entity_registry.py", line 1312 in _data_to_save
  File "/usr/src/homeassistant/homeassistant/helpers/storage.py", line 541 in _write_data
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58 in run
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 92 in _worker
  File "/usr/local/lib/python3.12/threading.py", line 1010 in run
  File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

https://github.com/home-assistant/core/blob/0c44051d2ab8a78d8b979aaf950f5dc1c5c5bfa4/homeassistant/helpers/json.py#L158

Current thread 0x00007fb7989fbb38 (most recent call first):
  File "/usr/src/homeassistant/homeassistant/helpers/json.py", line 158 in _orjson_bytes_default_encoder
  File "/usr/src/homeassistant/homeassistant/helpers/json.py", line 188 in save_json
  File "/usr/src/homeassistant/homeassistant/helpers/storage.py", line 544 in _write_data
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58 in run
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 92 in _worker
  File "/usr/local/lib/python3.12/threading.py", line 1010 in run
  File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

https://github.com/home-assistant/core/blob/0c44051d2ab8a78d8b979aaf950f5dc1c5c5bfa4/homeassistant/helpers/json.py#L158

Current thread 0x00007f7bccdffb38 (most recent call first):
  File "/usr/src/homeassistant/homeassistant/helpers/json.py", line 158 in _orjson_bytes_default_encoder
  File "/usr/src/homeassistant/homeassistant/helpers/json.py", line 188 in save_json
  File "/usr/src/homeassistant/homeassistant/helpers/storage.py", line 544 in _write_data
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58 in run
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 92 in _worker
  File "/usr/local/lib/python3.12/threading.py", line 1010 in run
  File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

Current thread 0x00007f4019bffb38 (most recent call first):
  File "/usr/src/homeassistant/homeassistant/helpers/json.py", line 158 in _orjson_bytes_default_encoder
  File "/usr/src/homeassistant/homeassistant/helpers/json.py", line 188 in save_json
  File "/usr/src/homeassistant/homeassistant/helpers/storage.py", line 544 in _write_data
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58 in run
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 92 in _worker
  File "/usr/local/lib/python3.12/threading.py", line 1010 in run
  File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

https://github.com/home-assistant/core/blob/0c44051d2ab8a78d8b979aaf950f5dc1c5c5bfa4/homeassistant/helpers/json.py#L158

Current thread 0x00007f0eae8ffb38 (most recent call first):
  File "/usr/src/homeassistant/homeassistant/helpers/json.py", line 158 in _orjson_bytes_default_encoder
  File "/usr/src/homeassistant/homeassistant/helpers/json.py", line 188 in save_json
  File "/usr/src/homeassistant/homeassistant/helpers/storage.py", line 544 in _write_data
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58 in run
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 92 in _worker
  File "/usr/local/lib/python3.12/threading.py", line 1010 in run
  File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

[Petro31]

Got another

Current thread 0x0000ffff4efdf758 (most recent call first):
  File "/usr/src/homeassistant/homeassistant/helpers/json.py", line 158 in _orjson_bytes_default_encoder
  File "/usr/src/homeassistant/homeassistant/helpers/json.py", line 188 in save_json
  File "/usr/src/homeassistant/homeassistant/helpers/storage.py", line 544 in _write_data
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58 in run
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 92 in _worker
  File "/usr/local/lib/python3.12/threading.py", line 1010 in run
  File "/usr/local/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/local/lib/python3.12/threading.py", line 1030 in _bootstrap

This one is a home assistant blue.

[bdraco]

So thats one is aarch64

[bdraco]

Looks like this commit has an aarch64 and x86_64 only path

58a73f3#diff-abdceb17520cb5accf57f4d4eed04d97b84d89b341ce6c14071b9b4f79390c5cR115
58a73f3#diff-abdceb17520cb5accf57f4d4eed04d97b84d89b341ce6c14071b9b4f79390c5cR116

[bdraco]

Music Assistant reported the same issue and I think they went from 3.10.0 to 3.10.1 so that commit being the source tracks with the timeline

[bdraco]

music-assistant/hass-music-assistant#2110 music-assistant/hass-music-assistant#2142

[bdraco]

Looks like the problem is probably fixed in 3.10.2. Thanks

Unfortunately to looks like armv7 support is gone so we are stuck on 3.9.15

[bdraco]

I tested 3.10.3 quite a bit today, and I couldn't repro the seg fault (I also couldn't repro it before on 3.10.1 though)

Everything does still appear to be working as expected on armv7 and wheels build ok for it.

[bdraco]

We should get confirmation that the problem is fixed on May 29/30 from the next Home Assistant beta.

[bdraco]

Awaiting Home Assistant 2024.6.x beta to provide confirmation the seg fault issue is solved

[ijl]

Ok, thanks for the report. I think this was the off-by-one fixed in 3.10.2. That also dropped arm7 wheels due the CI job breaking and not really being something I can fix. If someone if motivated enough to do a just cross-compile or cross-compile and test in QEMU job that would be welcome, I just can't. Please reopen if not fixed.

[bdraco]

Thanks for following up

Home Assistant is successful building wheels for armv7 (and they work) on newer versions. I'll see about getting a PR together to add back armv7 later this week.