QUIC Concurrency API Implementation

[hlandau]

Status: In progress — debug and test. Not for review yet unless people are interested. Just putting this up for now as a preview of how the solution is going to work. Bit of a change as I usually don't put up drafts, but I figured I'd give people a view of what's going on.

---

This is a partial implementation of just enough of #24256 to fix #24166 intended for 3.4. It introduces necessary concurrency model concepts to allow applications to opt in to reliable blocking functionality if they want it, and allows them to forego it if they do not.

It adopts the concept of a QUIC Domain SSL object which was previously discussed in the Server API design document but at that time deferred for possible (post Server MVP) future implementation. Explicit concurrency control makes having explicit control over an event domain, and the ability to have multiple listeners in the same domain, highly desirable, and the internal architecture for this was already implemented anyway.

A new API is added to allow configuring a desired concurrency model.

[nhorman]

why WSAsocketA here? Seems like you could just use BIO_socket in both cases, no?

[hlandau]

This allows WSA_FLAG_NO_HANDLE_INHERIT to be set, which is the equivalent of O_CLOEXEC... BIO_socket does have an unused options flag so we could add this functionality to BIO_socket, of course, though that's a public API augmentation.

[nhorman]

You need to lock this mutex prior to reading/writing rtor->signaled_notifier, and unlock it again after the thread you are notifying has updated signaled_notifier. I believe if the mutex isn't owned by the calling thread at the time of the underlying pthread_cond_wait call, EINVAL is returned here:
https://pubs.opengroup.org/onlinepubs/7908799/xsh/pthread_cond_wait.html

[hlandau]

Yes — this code is always called with the mutex held. Incidentally, poll_two_descriptors unlocks the mutex before calling poll(2) and then relocks it.

[nhorman]

I'm sorry, I see where poll_two_descriptors unlocks/relocks the mutex several times, but I never see where that mutex is first acquired.

Is the reactor mutex assigned in ossl_quic_reactor_init aliased from the qctx mutex that is also used in qctx_lock? That would explain why I can't find where rtor->mutex is explicitly locked.

Incidentally, while looking at this I noticed that all your usages of mutexes in QUIC are conditional on OPENSSL_THREADS being defined, which is good, but the above call to ossl_crypto_condvar_wait is not, implying that in a no-threads build, this code will pass a NULL pointer to ossl_crypto_condvar_wait, which I believe will cause -EINVAL to be returned. Not sure if thats a problem in an unthreaded build, but I thought I would point it out.

[hlandau]

Is the reactor mutex assigned in ossl_quic_reactor_init aliased from the qctx mutex that is also used in qctx_lock? That would explain why I can't find where rtor->mutex is explicitly locked.

Yes. The reactor does not allocate or own its own mutex, but uses the mutex donated from the quic_impl.c code. This is necessary so that it can unlock the mutex during the poll(2) call and then reacquire it.

[hlandau]

Incidentally, while looking at this I noticed that all your usages of mutexes in QUIC are conditional on OPENSSL_THREADS being defined, which is good, but the above call to ossl_crypto_condvar_wait is not, implying that in a no-threads build, this code will pass a NULL pointer to ossl_crypto_condvar_wait, which I believe will cause -EINVAL to be returned.

Good point. We need to have better CI coverage here...