bats: add support for deteriministic --shuffle

[cyphar]

There are many cases where tests accidentially depend on one another,
and bats users might want to ensure their test suite does not have this
behaviour. While --jobs gives you some randomness in the execution,
--shuffle allows for deterministic ordering in order to re-test a
particular case (by setting BATS_SHUFFLE_SEED) and also shuffles all
tests entirely.

In order to avoid a dependency on GNU coreutils (for OS X users), we
can't use shuf(1) and instead have to resort to a fairly contorted usage
of paste(1). On the plus side is means we can be sure that the ordering
won't change with new shuf(1) versions.

This is a follow-up of #172.

Signed-off-by: Aleksa Sarai cyphar@cyphar.com

• I have reviewed the Contributor Guidelines.
• I have reviewed the Code of Conduct and agree to abide by it

[cyphar]

This still needs some tests -- though I'm a little cautious about how we're going to test the seed values in a way that doesn't cause us problems in the future (if awk's srand starts producing different values). I will work on that when I have some time.

[cyphar]

As an update, it looks like old bash versions don't like the arrays_equal code. I will need to take a look at this...

[jasonkarns]

I'm 👍 on the feature. Haven't looked at the implementation, though. But before we settle too hard on the interface (the flag and var names), can we put together a quick survey of what other tools are using for their CLI flags/options/var names? That way we can follow any existing norms, however, slight they may be?

[cyphar]

@jasonkarns As far as I'm aware there aren't many other testing tools which provide something like this. @sublimino mentioned this in #171. Ruby has --seed and random ordering is the default, so I could add another command-line flag if you like (but the default ordering is going to be in-order to avoid breaking people's tests).

[jasonkarns]

@cyphar Virtually every testing tool that I'm aware of, which supports random execution, also supports re-running a prior random order deterministically.

RSpec:

--order defined
--order rand
--order rand:123
--seed 123 # same as --order rand:123

Minitest is random by default and accepts seed via -s, --seed or SEED env var.
(with an in-test helper to force non-random order: i_suck_and_my_tests_are_order_dependent!)

Cucumber uses --order random and --order random:SEED

In JavaScript/node land:

Mochajs doesn't officially support random order, but the "heavily used" patch uses --random SEED.

Jasmine has --random=true|false and --seed=SEED

Tap, Jest, Ava, Cypress don't seem to support true randomization nor determinism with a seed; though they support parallelization of the suite so there's some implicit randomicity but without the ability to re-run that particular order. (And they can both force serial runs, which of course isn't the same as deterministically re-running a random order.)

The rest of the common JS tools (tape, teenytest, nightwatch, qunit) don't seem to support randomization at all.

I would be curious to see what features the top 2 or 3 testing tools of python, php, perl, go, java, .net support. At the very least --seed so far seems to be a de facto standard for the seed.

[jasonkarns]

Personally, I do like cucumber, rspec, and mochajs' dual use of a single flag for both enabling randomization and providing the seed (since a seed without random ordering is meaningless).

--order rand[:seed] is a bit heavy on the parsing. --random [SEED] is a simple compromise (the seed argument is optional).

--seed SEED on its own implying randomization seems to be the most straightforward of all the options. It follows the "convention" that all of these tools use for --seed, and since --seed without randomization is meaningless, it makes sense to me that the presence of the flag also triggers the shuffling. The downside, is that we would still need a separate flag to trigger shuffling without a seed.

[cyphar]

I will go with --random [SEED] then. My point was that:

Virtually every testing tool that I'm aware of, which supports random execution, also supports re-running a prior random order deterministically.

Is not a large number of tools, not that no tools which support random execution support re-using the same order (in fact I would argue the opposite because it's such an obvious thing to need).