Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests that compete for globally shared resources are flaky when run in parallel #44972

Open
jgraham opened this issue Mar 7, 2024 · 2 comments

Comments

@jgraham
Copy link
Contributor

jgraham commented Mar 7, 2024

Originally filed as https://bugzilla.mozilla.org/show_bug.cgi?id=1866605

In that example we have two tests that use the clipboard. When run in parallel the tests both writing to and reading from the clipboard have an obvious race condition which causes intermittent failures. Our usual approach of partitioning the state doesn't work here because it's OS-level global state; partitioning would be equivalent to requiring tests to be run single-process-per-OS-instance.

To solve this the obvious abstract requirement is some way to annotate which tests are known to require the same shared state. For example:
<meta name=state content=clipboard>

Functionally it would then be up to the runner to ensure that multiple tests declaring the same state requirements don't run in parallel. This could be achieved by each type of state being associated with a mutex that must be acquired before starting the test (should be OK if the number of tests is small and contention is unlikely) or by scheduling such that those tests are never run in parallel.

Requirements wise a question is whether there's a case where we want vendor-specific global state annotations in expectation metadata files. This could be required if a certain browser has implementation-specific global state (e.g. a case where using the media stack from multiple processes at once causes issues).

@jgraham
Copy link
Contributor Author

jgraham commented Mar 7, 2024

CC @jcscottiii @gsnedders

@gsnedders
Copy link
Member

cc @JonWBedard, given this has been a broader problem for WebKit (and being able to use the same solution for both might be nice!)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants