Provide means to use RandomizedTesting without the test runner.

[dsmiley]

I work on a project that depends on lucene-test-framework for its wonderful test utilities to validate use of text analysis components. The project does not use randomizedtesting's test runners. I want to use some test utilities that I cannot invoke now because they directly/indirectly invoke randomizedtesting which fails because there is no context:

java.lang.IllegalStateException: No context information for thread: Thread[id=1, name=main, state=RUNNABLE, group=main]. Is this thread running under a class com.carrotsearch.randomizedtesting.RandomizedRunner runner context? Add @RunWith(class com.carrotsearch.randomizedtesting.RandomizedRunner.class) to your test class. Make sure your code accesses random contexts within @BeforeClass and @afterclass boundary (for example, static test class initializers are not permitted to access random contexts).

Asking users/projects to a specific test runner can be an awkward constraint. For some of my tests, I can use your runner, for some, I cannot because another is used.

When I look at RandomizedContext, there isn't sufficient access for me to manually manage it's lifecycle outside of your runner. I would like this added and/or a JUnit Rule as an option. Such a Rule wouldn't bring all the baggage that RandomizedRunner has -- no thread leak detection. Just provide what this project's namesake is -- randomness. That's all.

[dweiss]

What is it that threw this exception?

I am really hesitant about exposing too many internals for a number of reasons: once it's out, it's additional maintenance but mostly randomization without proper control and setup (done in the custom runner) is against the philosophy that I tried to convey with this project.

If you don't isolate threads between threads then reproducibility is gone. Also, the default runner makes a number of tricks to ensure repetitions work properly, different seeds are picked for each run, test descriptions are formatted in a way that is IDE-friendly... There is probably more.

If you only need test utilities then many of them accept an external Random instance [1] - then you don't need the context to be initialized. Some checks in the runner can be turned off via annotations if they're annoying. I would need to see a compelling reason for this though.

[1] https://github.com/randomizedtesting/randomizedtesting/tree/master/randomized-runner/src/main/java/com/carrotsearch/randomizedtesting/generators

[dsmiley]

I understand there is a lot of value in totally embracing all of what RandomizedTesting provides. I just think there should be a light-weight "Rule" based option for those that want/need something lighter weight, even if it's less rigorous (gotchas to repeatability -- may not be as repeatable in certain cases). Put differently, I don't think it needs to be all or nothing.

BaseTokenStreamTestCase.assertAnalyzesTo. Indeed, you are correct that I could put a 3-line method of the same name into my project and get a random number to pass to checkAnalysisConsistency. That said... I would miss what RandomizedTesting provides -- repeatable randomness.

[dweiss]

I don't think repeatable randomness can be implemented with a rule - that's the whole point I tried to make. Maybe with the new JUnit5 infrastructure it's different, I didn't look into this yet.