HTTP/2 Support and shared clients

[TilBlechschmidt]

Even though the official Selenium Grid does not support HTTP/2 just yet, there is a distinct possibility that it will eventually do so. Additionally, there may be other grid solutions out there that do support it already (shameless plug 😉).

Especially when running multiple tests in parallel across multiple browser sessions, using a single HTTP/2 connection to the Grid with request multiplexing can help reduce network load and latency. Thus I'd really like to see this feature.

Regarding the implementation, it should be somewhat straightforward. We need a way to share the Reqwest Client structure between multiple WebDrivers and call the http2_prior_knowledge function on the Client to force HTTP/2 (it does not appear to negotiate it on its own).

[TilBlechschmidt]

Sharing the Client struct should be relatively easy since it internally uses an Arc to share the connection already so we just have to clone it when creating further WebDriver instances.

[stevepryde]

It probably makes sense to wait until this is supported by Selenium so that it can actually be tested 🤔.

Meanwhile if we want to share the same client across multiple WebDrivers then all we really need is to clone the sender channel used to send webdriver requests. All requests will still be done in series though, not parallel, due to the way the channel is consumed only via a single thread. If this isn't sufficient then a more elaborate solution would be needed.

[stevepryde]

Also check out https://github.com/stevepryde/xenon - it works as a lightweight selenium server (but just as fast) and may be possible to add support for HTTP/2. If that works then it might be an option to test against as well.

[TilBlechschmidt]

You can test it by pointing it at a deployment of this* 🤷‍♂️

It'd obviously increase test complexity by requiring yet another grid software deployed but would work.

_{*The HTTP/2 feature is not yet in a release but that is coming soon — I just need to patch up a few things that broke by migrating to h2.}

[TilBlechschmidt]

Also check out https://github.com/stevepryde/xenon - it works as a lightweight selenium server (but just as fast) and may be possible to add support for HTTP/2. If that works then it might be an option to test against as well.

Oh look another project that noticed the immense resource consumption of Selenium Grid and used Rust for a more efficient implementation 😆

Jokes aside, thanks for the link!

[stevepryde]

You can test it by pointing it at a deployment of this* man_shrugging

Ha! This looks awesome, although I need to read more of the docs to see what it can do. And yes I routinely find selenium hub consuming more than 1 core on my PC for what should be effectively a thin proxy. Not to mention consuming upwards of 300MB of memory constantly. Good ol' JVM. Of course browsers are another story :(
If you're able to test against your project then that works too :)

[TilBlechschmidt]

Just to give some spoilers (although this is getting more and more off-topic):

We were able to run ~700 browsers behind a single WebGrid instance across 41 K8s nodes. All with screen recordings running, obviously 😜

[lucaswxp]

@TilBlechschmidt Just out of curiosity, what kind of machines were those node? Memory/CPU/Datacenter etc.

[TilBlechschmidt]

@TilBlechschmidt Just out of curiosity, what kind of machines were those node? Memory/CPU/Datacenter etc.

I can't remember exactly but they were running in the Google Compute Cloud (GKE) and must've been around 8-16 cores each with probably 32GB of RAM. However, it really depends on your type of workload how much power and memory you need. Browsers typically have a large startup overhead but once they are running, the requirements are pretty modest. In the future, I plan to do some more detailed analysis based on real-world workloads.

At some point, I'll have to write a blog post about all this but I'm afraid it'll be some time until that happens 😄

On-premise we've been running 50-100 browsers on a 40 core, 256GB machine which was not anywhere near its capacity limits -- again all depends on workload as this was while running a stress test designed for WebGrid and not the browsers. Then again, it ended up being more of a stress test for our K8s Control plane ...

[TilBlechschmidt]

If you have other questions @lucaswxp, we should probably move the discussion over to the appropriate repository though to prevent this issue from going off-topic.

Either the GitHub Discussions or the Discord server would be a good place 🙂