Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shorten test runs using LPT algorithmn #77

Open
eraserhd opened this issue Jan 6, 2020 · 2 comments
Open

Shorten test runs using LPT algorithmn #77

eraserhd opened this issue Jan 6, 2020 · 2 comments

Comments

@eraserhd
Copy link
Contributor

eraserhd commented Jan 6, 2020

Since running test in parallel is exactly the "Multiprocessor scheduling problem", sorting the jobs by last known run time, then running longest first, should shorten the test time significantly (and also make run time more stable).

Proposal: We add a defonce atom with a map from test symbol to last known runtime in milliseconds. Sort the tests by this with some arbitrary default value if missing before assigning to threads, and record run time to the atom as tests complete. The first test run will effectively work the same way as it does today, but subsequent test runs should complete earlier. As the atom is just a heuristic, it is OK for it to be out of date.

I'm willing to do the work here, but does this sound good?

@eraserhd
Copy link
Contributor Author

eraserhd commented Jan 7, 2020

This'll require significant rework of the runner.

The process of running each namespace in a thread, and the namespace thread submitting each test to a fixed executor, should make the test order somewhat non-deterministic (even when the seed is fixed), but it also doesn't allow sorting through namespaces. So it would need one thread to order and one thread to reassemble the jobs and make NS end reports.

@eraserhd
Copy link
Contributor Author

eraserhd commented Jan 9, 2020

Some rough code to test the hypothesis shows about 8% improvement on a complicated test suite, which is nice, but not worth the investment for me at the moment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant