wifi fails with ECONNABORTED when microdot webserver is stressed

[heuj]

Hey,
have an application microdot webserver running on MicroPython v1.22.2 on 2024-02-22; ESP32C3 module with ESP32C3.

The application provides both a html page and rest api. When accessing only the api (only tiny data) runs rock solid.
However very frequently when a browser reloads the whole page (some js and html files) the application fails in the following way:

microdot.py", line 641, in write
File "asyncio/stream.py", line 1, in stream_awrite
File "asyncio/stream.py", line 1, in write
OSError: [Errno 113] ECONNABORTED

After this I can see that wifi isconnected() reports False and needs a few secs to reconnect.
Tried on different APs behavior is same.

Anyone seen behavior like this?

edit: osdebug shows:I (72530) wifi:bcn_timeout,ap_probe_send_start
I (75030) wifi:ap_probe_send over, resett wifi status to disassoc
I (75030) wifi:state: run -> init (c800)
I (75030) wifi:pm stop, total sleep time: 31598179 us / 72447704 us

I (75030) wifi:idx
I (75030) wifi:new:<1,0>, old:<1,0>, ap:<255,255>, sta:<1,0>, prof:1
I (75030) wifi: STA_DISCONNECTED, reason:200:beacon timeout
I (75040) wifi:new:<1,0>, old:<1,0>, ap:<255,255>, sta:<1,0>, prof:1
I (75040) wifi:state: init -> auth (b0)
I (75040) wifi:state: auth -> assoc (0)

[peterhinch]

There is some information in this ESP doc. My guess is that the AP is so busy that it stops sending beacon messages and the client aborts the link.

[heuj]

Thank you for your reply! That is very interesting, would not have expected the the root cause in this direction!
The APs I tried with are pretty much std, wouldn't there be a lot more reports on this if this behavior is relatively unrelated to my application?

Found a discussion (for normal ESP32) which seem to go into the direction of memory allocation failure and/or flash read/writes blocking the detection of beacons: https://www.esp32.com/viewtopic.php?t=6242
Could this be possible for ESP32_C3 as well?

Maybe do you have suggestions on how to debug this?

[heuj]

Did the following experiment: Minimal example on ESPC3: Wifi, Microdot, 1 large file which can be downloaded.

1. Get file with python script from PC in a loop --> stable
2. Get file with python script from PC in a loop, but in 5 threads --> wifi of ESP quickly fails microdot throws ECONNABORTED

Further experimentation indicates that if microdot tries to send multiple Responses while transmission takes a while due to file size the error is far more likely.

Unsure if this is due to the AP stopping beacons or some ESP internal problem.

As a quick workaround I added a lock forcing microdot to handle the requests one by one including time needed for transmission. Not nice but improves stability significantly.

[Carglglz]

@heuj esp-idf is running out of heap see micropython/micropython-lib#851 (comment)

[heuj]

@Carglglz Thanks for the hint! Would fit that my app never fails on first few page reloads, but fails with high probability after a ~const amount of quick page reloads.
I can change my app to make failure less probable but so far I never know how much margin is left. Is there a reliable way of detecting idf out of memory condition so I can test my app under load? There seems no out of memory error here and gc.mem_free and esp32.idf_heap_info only show free mem is decreasing but app fails way before any of those numbers get close to zero.

[Carglglz]

There seems no out of memory error here and gc.mem_free and esp32.idf_heap_info only show free mem is decreasing but app fails way before any of those numbers get close to zero.

Yes that is the thing, as the docs says

it is very dangerous to run out of heap memory, as this will cause ESP32 an "undefined behavior".

which in this case is that the network stack will stop.

What worked for me was allocating a buffer of 256 bytes and then load chunks of the data/file into that buffer and send it.
So preallocation and garbage collection to avoid MicroPython heap growth too much is the way to go.

Is there a reliable way of detecting idf out of memory condition so I can test my app under load?

I think the amount of memory needed depends of the number of connections and the chunk size being sent.
see also peak-wi-fi-dynamic-buffer