Support shuffling / randomized test runs

[twhiting]

Checklist

• Feature request has a meaningful title
• I have searched the existing issues. See all issues
• I have tested using the latest version of Pester. See Installation and update guide.

Summary of the feature request

Google test has a feature where tests can be shuffled randomly per test run. This is beneficial in integration tests scenarios when one test might fail as a result of another previously ran test. Frankly it helps find obscure bugs.

How should it work?

See google test shuffle for a good description: http://google.github.io/googletest/advanced.html#shuffling-the-tests

[nohwnd]

Interesting. That would a bit difficult (and inefficient) to provide, because Pester relies on execution that is "top-down", so if we mix tests that come from 2 different files we have to "restore" the state for each test. And run all setups and teardowns again, breaking some of the life cycles that people probably rely on. This would be specially painful for mocks I think.

I am not against it completely, I would just like a bit more data showing why doing this is worth it.

[fflaten]

Yes it would be limited to shuffling tests (It) inside the current block (Context/Describe). Is that enough to be useful?

Shuffling containers/files is also an option, though that can already be achieved manually.

[twhiting]

Yes google test shuffles within a "test suite" which is the behavior that @fflaten describes.

[twhiting]

It is hard to explain why it's worth it without being super abstract. I for example have a powershell module that interfaces with a windows service. It is possible that test C can cause a state within the service that test A exposes. It is VERY hard to track down these cases without a reliable randomizer.

The second part of this is needing to be able to run the tests again in the same shuffled order.

Google test can shuffle based on a seed. So say in CI a test fails when shuffled with seed X. I can then manually trigger a new run with seed X and get the same exact test run order for all tests.

[twhiting]

For a good description of the seed mechanism see the google test link I posted above.

[nohwnd]

Makes sense. If we randomize within the same container, in a top-down way (that is we randomize on each level, but dont jump up and down), then it is hardly any change from the current way of running tests, all you need to do is shuffle the tests in the discovered tree (in a deterministic way). I think there is even an “order” list already in the discovered tree.

@twhiting do you want to make a PR for this? Even if its just a proof of concept.

[twhiting]

@nohwnd i'd love to get to the feature myself but realistically it'd be weeks/months. So if this is something you are interested in feel free to jump on it!

Thanks.