When disconnecting from network before request is completed microdot dies hard

[mirko]

This is happening on ESP32 with micropython - so "network" and "wifi" are used interchangeable.
When I'm disconnecting from WiFi within a microdot-handler, microdot fails live below and does not recover from this (as in: does not serve requests anymore after reconnect):

I (802826) wifi: STA_DISCONNECTED, reason:201
Task exception wasn't retrieved
future: <Task> coro= <generator object 'serve' at 3ffd8c00>
Traceback (most recent call last):
  File "uasyncio/core.py", line 1, in run_until_complete
  File "microdot_asyncio.py", line 268, in serve
  File "microdot_asyncio.py", line 336, in handle_request
  File "microdot_asyncio.py", line 143, in write
  File "uasyncio/stream.py", line 1, in stream_awrite
  File "uasyncio/stream.py", line 1, in write
OSError: [Errno 113] ECONNABORTED
Task exception wasn't retrieved
future: <Task> coro= <generator object 'serve' at 3ffd8910>
Traceback (most recent call last):
  File "uasyncio/core.py", line 1, in run_until_complete
  File "microdot_asyncio.py", line 268, in serve
  File "microdot_asyncio.py", line 336, in handle_request
  File "microdot_asyncio.py", line 157, in write
  File "uasyncio/stream.py", line 1, in stream_awrite
  File "uasyncio/stream.py", line 1, in write
OSError: [Errno 113] ECONNABORTED

When shortly afterwards reconnecting back to WiFi and the device being reachable again, microdot's state doesn't allow to accept further connections anymore.

While I see one shouldn't purposely drop the layer-2 connection within the handler, I do believe this might happen in the wild and microdot should recover and - once l2-connection is re-established - continue serving data.
If that's intended behaviour and one is supposed to restart microdot oneself after l2-reconnects, maybe a note within the docs would help and/or an example on how to best deal with such a situation.

[miguelgrinberg]

How did you reconnect the device after the failure? Microdot does not handle the networking, it assumes that this is managed by the application. My guess is that the server just errored when the network disappeared, so you could catch the exception, and when you can, reconnect and start the server again.

[mirko]

Use case is, I'm intentionally disconnecting and reconnecting WiFi (to be precide, switching WiFi networks, but staying in the same IP space)..
It's triggered by user action via the webinterface.
And I'd love to see the AJAX request - triggering the WiFi switch/reconnect - being completed.
When catching the WiFi-disconnect/exception and restarting microdot, the entire TCP-session will be lost.
I was hoping for a way microdot being able to deal with this "hiccup" and continue trying to (re)send its response, but dealing with l2-reconnects is probably (way) out of its scope.

[miguelgrinberg]

And I'd love to see the AJAX request - triggering the WiFi switch/reconnect - being completed.

If the device has no network there is no way to tell it to reconnect, so while this sounds great in practice, it is not possible. The device can attempt to reconnect on its own in a retry loop, I guess, but I want this project to be independent of the underlying networking so I don't see it likely that something like this will be added.

I was hoping for a way microdot being able to deal with this "hiccup" and continue trying to (re)send its response

This is not really in scope for this project. What you want is a network stack that is able to handle disconnections and reconnections internally, without affecting the applications that use it.

[sornen]

I have this problem as well when wifi disconnects but when I try to catch the exception say around ws.receive I get other exceptions which did not occur before. Now when the browser is refreshed or the client disconnects there are additional exceptions such as ENOTCONN, ECONNRESET, and 32 which micropython does not label for the esp32, but is EPIPE.

Also [Errno 113] is EHOSTUNREACH not ECONNABORTED.

[miguelgrinberg]

I have not tried this myself, but if you catch the exception generated when the network goes down in a HTTP or WebSocket handler you will not be able to do anything to recover from it, so I'm not surprised you get more exceptions later.

The correct thing to do is to catch the exception as it bubbles out of the server (i.e. at the app.run() or app.start_server() level). Then when the network is back you restart the server and I think everything should work again, without needing the big hammer of a power cycle.

[sornen]

I tried this and it still gives an error

Traceback (most recent call last):
  File "microdot_asyncio.py", line 370, in dispatch_request
  File "microdot_asyncio.py", line 443, in _invoke_handler
  File "microdot_asyncio_websocket.py", line 97, in wrapper
  File "wshandler.py", line 21, in readws
  File "microdot_asyncio_websocket.py", line 17, in receive
  File "microdot_asyncio_websocket.py", line 36, in _read_frame
  File "uasyncio/stream.py", line 1, in read
OSError: [Errno 113] ECONNABORTED

I did find that the websocket server is still running after the error and once the esp32 reconnects to wifi the websocket server can reconnect to websocket clients without a hard reset. Tried this 8 times and the esp32 keeps running and accepting new websocket connections once wifi is reconnected. There seems to be no other problems, although I'm a bit concerned about memory leaks from the error. I think the problem may be with lwip in micropython rather than microdot.

So in summary all that needs to be handled is reconnecting to the wifi by monitoring sta.isconnected() if a wifi connection is lost, and then reconnecting once the wifi is up. Nothing needs to be done about the websocket server.

[maxi07]

I am dealing with the same issue. I am starting the webserver with app.run(debug=True) and from the webinterface I am selecting a new SSID and connect to such successfully. I am trying to catch the whole app.run block but with no success. My other idea was to shutdown the server before attempting to connect to a new SSID, which I did with request.app.shutdown when receiving the new credentials from the webserver. This command itself did not report any errors but I have the feeling it does not shutdown correctly as I am still receiving the same File "uasyncio/core.py", line 1, in run_until_complete error. Any idea?

[sornen]

A method for getting around the ECONNABORTED OSError is to monitor the network.status() for STAT_BEACON_TIMEOUT condition. This can be used to hard reset the device so that it is back in a known configuration. Since the ECONNABORTED OSError only occurs when a client is connected and the wifi router goes down it is hoped that this is reasonably rare event. Here is example code.

import machine
import network
import uasyncio

async def connector():
    sta = network.WLAN(network.STA_IF)

    while True:
        if sta.status() == network.STAT_BEACON_TIMEOUT:
            machine.reset()

          .... rest of code to connect to network

        await uasyncio.sleep(1)

[mirko]

Not sure if it helps in this case, but what I noticed is there's a list of network errors in microdot defined to be ignored ("muted").
Adding 113 to those helped me getting rid of uncaught and uncatchable CONNRESET exceptions on ESP32.

I simply did so without modifying microdot by:

import microdot_asyncio
# extending socket-errors
microdot_asyncio.MUTED_SOCKET_ERRORS.append(113) # 113 = ECONNRESET (at least on ESP32)

[miguelgrinberg]

Strange, because ECONNRESET should be 104, which is already included in the list of errors to mute.

[mirko]

Also in lwip/errno.h ECONNRESET is 104. Maybe I mixed something up in the comment, but it solved my problem.
What's 113 in lwip/errno.h, though, is: EHOSTUNREACH. Maybe that was my issue.