Add blocking progress mode to Python async

[pentschev]

Implements the blocking progress mode (UCX-Py default), which was still not implemented in UCXX.

[wence-]

I think the core of this looks fine, but I am reminded that this epoll_wait issue with asyncio doesn't actually quite work correctly with the code we are using (see discussion here rapidsai/ucx-py#888)

[wence-]

Although I think I understand the constraint, I am not sure what it means for the "next state" to be epoll_wait. Surely there can be arbitrary non-ucx tasks?

[pentschev]

I'm gonna be honest and say my understanding here is also a bit fuzzy, and as you noted yourself this "doesn't work" (except when it does), with the original being adapted from https://stackoverflow.com/a/48491563. In my understanding, what epoll_wait refers to here is the socket state, which is there solely to provide a mechanism to prevent asyncio from running out of "ready" tasks, so epoll_wait will ensure that if nothing useful happens in the event loop, asyncio will still be awaken at some point to allow the loop to reevaluate.

[wence-]

Who does the epoll_wait call?

[pentschev]

As per my #116 (comment), I believe this is related to the sockets.

[pentschev]

I think the core of this looks fine, but I am reminded that this epoll_wait issue with asyncio doesn't actually quite work correctly with the code we are using (see discussion here rapidsai/ucx-py#888)

Yes, I remember that. The purpose of this is not necessarily to be used long-term, but rather to have a fallback to original UCX-Py behavior. This will allow us testing as close as possible with what UCX-Py did in the past, and we may later deprecate/remove this if we are confident we have a better option (e.g., thread progress mode).

[wence-]

Thanks for the explanations!