Truely random test order with posibility for externaly given seed

[cardil]

By default junit executes test in default order that is based on method name hash code. In java hashcodes of string doesn't change between executions, so the order is random, but constant for given computer. That could lead to unpredictable errors.

Ref: https://github.com/junit-team/junit/blob/master/src/main/java/org/junit/internal/MethodSorter.java#L13

$ groovysh 
Groovy Shell (1.8.6, JVM: 1.7.0_79)
Type 'help' or '\h' for help.
---------------------------------------------------------------------------------------------------------------------------------------------------------
groovy:000> a = 'testSample'
===> testSample
groovy:000> a.hashCode()
===> 1710059740
groovy:000>
$ groovysh 
Groovy Shell (1.8.6, JVM: 1.7.0_79)
Type 'help' or '\h' for help.
---------------------------------------------------------------------------------------------------------------------------------------------------------
groovy:000> a = 'testSample'
===> testSample
groovy:000> a.hashCode()
===> 1710059740
groovy:000>

To fix this enchantment, there should be truly random order option to explicitly specify in test case using @FixMethodOrder.

I propose to follow Rspec --order random option implementation. It actually makes random order for execution but seed used for this is displayed and logged into reports. This gives developers a chance to execute the same execution with --order rand:3455. It guarantees that tests will be executed in the same order, therefore it's possible to hunt for execution order bugs.

Cite:

When you use --order random, RSpec prints out the random number it used to seed the randomizer. When you think you’ve found an order-dependency bug, you can pass the seed along and the order will remain consistent:

--order rand:3455

Example execution:

$ rspec test.rb --order rand

Randomized with seed 64539

testing
  should equal 5
  should contain 'test'

Finished in 0.02 seconds (files took 0.0092 seconds to load)
2 examples, 0 failures

Randomized with seed 64539

$ rspec test.rb --order rand

Randomized with seed 12834

testing
  should contain 'test'
  should equal 5

Finished in 0.02 seconds (files took 0.0088 seconds to load)
2 examples, 0 failures

Randomized with seed 12834

$ rspec test.rb --order rand:12834

Randomized with seed 12834

testing
  should contain 'test'
  should equal 5

Finished in 0.02 seconds (files took 0.0083 seconds to load)
2 examples, 0 failures

Randomized with seed 12834

Rspec random refenrence: http://blog.davidchelimsky.net/blog/2012/01/04/rspec-28-is-released/

[Jahhein]

Might I suggest knuth shuffle for determining run order?
info-
https://en.wikipedia.org/wiki/Fisher–Yates_shuffle

source code-
http://algs4.cs.princeton.edu/11model/Knuth.java.html

[ffbit]

From Test execution order

By design, JUnit does not specify the execution order of test method invocations. Until now, the methods were simply invoked in the order returned by the reflection API. However, using the JVM order is unwise since the Java platform does not specify any particular order, and in fact JDK 7 returns a more or less random order. Of course, well-written test code would not assume any order, but some do, and a predictable failure is better than a random failure on certain platforms.

From version 4.11, JUnit will by default use a deterministic, but not predictable, order (MethodSorters.DEFAULT). To change the test execution order simply annotate your test class using @FixMethodOrder and specify one of the available MethodSorters:

@FixMethodOrder(MethodSorters.JVM): Leaves the test methods in the order returned by the JVM. This order may vary from run to run.

@FixMethodOrder(MethodSorters.NAME_ASCENDING): Sorts the test methods by method name, in lexicographic order.

[Endron]

The problem here is that there are actually two test smells that are interconnected with each other:

1. Tests depending on the execution order
2. Non repeatable tests

Putting tests into a really random order would help with the first. As no test can predict what happened before he can not depend on any other test to have been executed before. By changing the order with each run you would also find tests accidentally depending on the execution order.

But if you change the execution order to random your tests might also become non repeatable. So if you run the tests on the same machine two times they may yield different results. This will most likely be because your tests are somehow coupled in some way. They should not be but they are. There is no easy way to deal with these results. What do you do if you try to deploy some software because everything was green all the time but than in the final test you by accident find the one execution order that brakes one test? Tests must ether pass or fail and they must do this repeatedly. (Or else someone will start ignoring failing tests and just rerun them to get them passing.)

This is most likely part of what let to the implementation as mentioned by @ffbit: It is a compromise to face both problems. By making the order unpredictable they made it hard for anyone to implement there tests to use it. By making it deterministic they ensured that test execution always comes to the same result.

[cardil]

Hey @Endron. Your point is good but please look closer in Rspec --order random option implementation. It actually makes random order for execution but seed used for this is displayed and logged into reports. This gives developers a chance to execute the same execution with --order rand:3455. It guarantees that tests will be executed in the same order.

Cite:

When you use --order random, RSpec prints out the random number it used to seed the randomizer. When you think you’ve found an order-dependency bug, you can pass the seed along and the order will remain consistent:

--order rand:3455

This is an example execution (you can see the seed is displayed in command line): https://travis-ci.org/coi-gov-pl/puppet-jboss/jobs/81936397#L556

[mrnichols76]

I just submitted a pull request adding this feature.
As this is my first time to contribute, your feedback is especially important. Thanks!

P.S. Sorry about the multiple commit notices. Being relatively new to GitHub, I didn't realize that merely pushing to my own repo would make it do that.

[cardil]

I think that this also can be done with PR #1130. The seed can be set externally by Maven and Gradle plugins

[odupuy]

We truly need a simple way as a new MethodSorters value and possibly as a global system property (RANDOM possibly but DEFAULT by default) to have a totally random order for any test not explicitly using one of the existing MethodSorters. Add some automatic seeding as needed to have a truly randomness without any special setting.
This will help a lot in finding all these badly written tests in all the legacy (or new) apps we work on.

[cardil]

I would like to say that if you are using Maven to run your tests you might be interested in this PR apache/maven-surefire#112 to Surefire/Failsafe plugin to introduce class level test randomization with seed.

[odupuy]

Thanks @cardil This will certainly come handy but when I run the test inside my IDE I usually run it directly so direct JUnit support is a must. I had a look at JUnit 5 as 4 is stale and this ticket should add some execution order in Junit 5.3