Removed unused sessions from SSL_CTX internal cache

[rzikm]

Discovered when playing with #101626 and #101552.

OpenSSL keeps an internal cache for TLS sessions so that it can automatically resume previous sessions created via same SSL_CTX objects. We do our own management of TLS session tickets on clients so that we don't try to resume sessions to different hostnames, and we keep exactly one SSL_SESSION object per hostname.

However, the internal OpenSSL cache still keeps track of all the SSL_SESSION objects. A single SSL_CTX object by default will be willing to cache 20 * 1024 objects by default, in my repro, this manifested as up to about 70 MB of "leaked" memory.

The fix is to simply synchronize both caches and remove the SSL_SESSION from SSL_CTX cache when we discard it.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

[wfurt]

and do we see positive improvements in memory use? I worry the tests we have don't quite reproesent realistic scenario.

[rzikm]

Do we know if that respects the 1x for each ticket? The report given to me claimed different memory use between 1.2 and 1.3 - all going to single server. But there were parallel requests and I'm not sure we have good test case for it.

While the change looks reasonable to me I feel we should make sure we handle all the TLS 1.3 cases as best as we can. I'm wondering if we have multiple TLS session to the same destination if we can get multiple tickets - each valid for one more resume. The current logic really maps to TLS 1.2 where we can simply keep the last session and use it later as many times as we want to.

few things:

• the "leak" presents only when there are parallel SslStream handshakes to the same host, what I observed and tried to fix is that we get multiple new session tickets from OpenSSL, but we keep only one. But even though we FreeSession the previous one, it still has a ref in the internal SSL_CTX cache, which should be now fixed by tha call to SslCtxRemoveSession
• even without this change, the TLS session is used multiple times if you start multiple TLS 1.3 handshakes in parallel, I checked and all these sessions get successfully resumed (at least running against Linux server), so I thought it better to preserve this behavior. I can reasonably enforce the single use constraint, I think we have TLS resume only for TLS 1.3 so it could be easy.

[rzikm]

and do we see positive improvements in memory use? I worry the tests we have don't quite reproesent realistic scenario.

Yes, with the change, the repro from #101552 presents stable OpenSSL memory utilization between iterations, as measured by #101626

[wfurt]

LGTM

[rzikm]

CI Failures are unrelated

[wfurt]

/backport to release/8.0-staging

[github-actions]

Started backporting to release/8.0-staging: https://github.com/dotnet/runtime/actions/runs/9036427971