Add support for JUnit5

[dadoonet]

Would be nice to add support for JUnit5.

[dweiss]

Would be. Feel free to provide a patch. :)

[dadoonet]

I started to see if I can do it (instead of opening an issue) but I've been blocked almost immediately with the ant part of the project. Some Java9 issues I guess... I might look at it indeed. :)

[dweiss]

To be honest I haven't had the need to use JUnit5, so my knowledge of it is very limited. I know they've changed the internals a lot, so it may be time-consuming refactoring. If there is not clear benefit I don't think I'll invest my time in doing this... there's just not enough of it anymore.

[dadoonet]

I can totally understand. That's what makes open source so funny: DIY! :p
I'll update the ticket if I make some progress. (Well, I do: building with Java8 works well)

[dadoonet]

@dweiss Correct me if I'm wrong but the maven plugin "just" delegates to the ant one.
If so I believe the first step is to create an junit5-ant module. But for that we need to wait for: apache/ant#60 to be merged and released.

[jprante]

I'm also interested in an update. JUnit 5 is a big step.

Randomized testing in JUnit5 might look like this https://github.com/qala-io/datagen/tree/master/junit5

Because I'm not interested in Maven or Ant, a pure Gradle solution is a viable alternative to me. Gradle 4.6 will support JUnit5 gradle/gradle#828 (comment)

There is a first step gradle/gradle#4116

Ordering sequence of tests might be important junit-team/junit5#13 (comment) was considered for JUnit 5.1 which was released yesterday https://github.com/junit-team/junit5/releases/tag/r5.1.0 but issue is still open.

Parallelized executions is another issue which is not settled in JUnit 5 junit-team/junit5#60 but would be very interesting

[alpar-t]

I'm also approaching this from the Gradle perspective.
At some point Gradle will make it impossible to replace the testing task as the plugin currently does.
Also by replacing the testing task, one misses out on features around build scan and reporting.
As part of elastic/elasticsearch#31496 I was looking into ways to be able to run the randomzied elasticsearch tests with Gradle. The major feature loss is around being able to randomized test case order, that for junit 4 is handled by Gradle, so Gradle could deal with the pseudo randomizing the test execution order - gradle/gradle#5760, but it supports junit 5 now and as @oehme points it, Gradle wants to build on top of junit platform going forward.

The only way I see to make the randomized testing Gradle plugin work past Gradle 6 is to use junit-vintage-engine to run with the Gradle Test task and junit platform and benefit from both Gradle and junit5 without code changes to the tests. To also get the randomized test order and other features, we would need to implement a test engine based on the junit-vintage-engine that has all the randomization features ? What do you think about that @dweiss ?

[dweiss]

I honestly don't know, haven't touched JUnit5 yet.

[Nick-Minutello]

I am using JUnit 5 and am sorely missing randomized testing.
It doesn't look like a small task to support Junit 5.
Is there any appetite to do it?
What are people using instead?

[dweiss]

You're correct - I don't think it's a small task, especially if you wanted to fit into JUnit5's new architecture more nicely... For now, I'm happy working with JUnit4 so there's no real itch to upgrade (and rewrite thousands of tests)...

[Nick-Minutello]

Do you even think it is possible without starting from scratch ?
I am wondering if it makes sense to start with a very cut-down implementation for junit5 & port features bit by bit?

[dweiss]

It depends on the direction, really. Most people think of JUnit5 in terms of Jupiter - the default TestEngine implementation. You could (easily?) port randomizedtesting's runner as a separate TestEngine but then none of jupiter's extensions would work, which is not much different than running JUnit5 tests with the vintage TestEngine...

A more ambitious plan would be to try to hack into jupiter to simulate what the custom runner currently does (primarily deterministic randomized context/seed providers for tests, hooks, etc., then thread leak detection and listeners, perhaps).

Last time I did some initial experiments [1], I couldn't make it work but I'm sure a lot has changed since then.

I think it'd be a fun project if you decide to spend some time on it. It's self-contained, which makes it even nicer. You can copy/paste or reuse whatever you need but you also don't have to - it's not about the implementation, it's about the functionality it has to offer. I'd start with the order in which the examples [2] are laid out - I tried to make it progress from easier/ more fundamental features to more esoteric ones.

[1] https://github.com/randomizedtesting/randomizedtesting/commits/junit5/
[2] https://github.com/randomizedtesting/randomizedtesting/tree/master/examples/maven/src/main/java/com/carrotsearch/examples/randomizedrunner