Core Concepts

What is tests randomization?

The main idea is this: a test case that runs in an exactly identical way every time it is run only covers that single execution path. Such tests are very good for verifying if any changes of behavior have happened when new code has been introduced (regression testing) or to assert on corner cases. These "fixed" tests do not bring any new insight into how the program behaves for previously unseen combinations of input arguments, components or environmental settings. And because for complex (or any) software such interactions are hard to predict in advance (think those buffer underruns, null pointers, etc.) running your tests on as many different input combinations as possible should over time increase the confidence that the software is robust and reliable.

The question how to implement the above concept of "different execution every time" and how to assert on conditions in such case can be solved in many ways. RandomizedRunner provides an implementation of java.util.Random which is initialized with a random seed that is reported (injected into a stack trace) in case of a test failure. So if a test fails it should be, at least theoretically, repeatable if started from the same seed.

For example, consider the code of add method:

/**
 * This method adds <code>a</code> and <code>b</code> and returns their sum.
 */
public static int add(int a, int b) {
  return a + b;
}

A fixed test scenario may look like shown below (we use assertEquals method from RandomizedTest class but otherwise the test's execution is always predictable):

@Test
public void fixedTesting() {
  // Note how we use superclass methods, RandomizedTest extends from
  // Assert so these methods are readily available.
  assertEquals( 4, Adder.add(2, 2));
  assertEquals(-1, Adder.add(0, -1));
  assertEquals( 0, Adder.add(0, 0));
} 

In a randomized test case, the execution will be different every time. For the add method we may randomize the arguments (within their contract bounds) and verify if the outcome satisfies some conditions. For this example, let's say the result of adding two non-negative integers shouldn't be smaller than any of the arguments:

@Test
public void randomizedTesting() {
  // Here we pick two positive integers. Note superclass utility methods.
  int a = randomIntBetween(0, Integer.MAX_VALUE);
  int b = randomIntBetween(0, Integer.MAX_VALUE);
  int result = Adder.add(a, b);
  assertTrue(result + " < (" + a + " or " + b + ")?", result >= a && result >= b);
} 

This test passes most of the time, but occasionally it will fail due to integer overflow.

Once a failing execution has been caught it's easy to repeat it (that's the whole point!). Note the first line of the stack trace, it contains the master randomization seed picked for the execution: 2300CE9BBBCFF4C8:573D00C2ABB4AD89. The first number if the "master" seed used in static suite context (class initializers, @BeforeClass and @AfterClass hooks), the second number is the seed derived from the master and used in a test context. To repeat the exact same failing execution we could either override seeds using system properties, as in:

-Drt.seed=2300CE9BBBCFF4C8:573D00C2ABB4AD89

or we could annotate the class/method in question and fix the seed to a particular value; for instance by adding an annotation to the class:

@Seed("2300CE9BBBCFF4C8:573D00C2ABB4AD89")

After doing so, we would be set for a debugging session to see what the cause of the problem was. The above example is part of a walk-through tutorial available in progressive difficulty.

See the javadoc and sources at GitHub