Rule org.junit.rules.TestWatcher not working, never signals failure

[meersdavy]

I am not sure if this is a JNario, or a JUnit issue.
I will try to explain what i noticed although it is not clear to me how everything fits together.

I try to use the TestWatcher to keep track of the scenario's that fail.
The problem is that the org.junit.rules.TestWatcher expects a Throwable to be thrown on failure which is not what happens when using a FeatureRunner, the ParentRunner.Run method swallows the throwable:

In JNario the rules are applied to the classBlock (FeatureRunner.classBlock),
which is a org.junit.rules.RunRules Statement
(as opposed to the methodBlock in BlockJUnit4ClassRunner.methodBlock).
This RunRules Statement has a base statement which is created in ParentRunner.childrenInvoker,
which iterates over every child of the ParentRunner and calls the abstract method runChild with the child as argument.

The ExampleGroupRunner.runChild method just runs the child runner.
The problem is that a throwable thrown by a child is swallowed by the ParentRunner, that throwable never reaches the TestWatcher which in turn always signals success, and never failure.

As i said, it is not entirely clear to me what happens.
It looks like the rule as used in a Feature differs from a testrule used in a standard JUnitTest.
They are not exactly the same since in JNario a feature is a combination of multiple standard JUnit tests, as a result the rule in a JNario Feature behaves more like a JUnit ClassRule (if i am not mistaken).

[sebastianbenz]

This looks like a problem with Jnario. I will look into it for the next release.

[sebastianbenz]

I had a look and it doesn't seem like there is a simple way to solve this. The problem is that some rules only make sense on scenario/class level, e.g. the TemporaryFolder rule. However, TestWatchers make only sense on a step level, because here are the exceptions thrown.

The cleanest solution would be to introduce a special ScenarioRule, which is specific to the semantics of Scenarios and Steps. I don't think that this will make it into the next release, but pull requests are of course welcome ;-).

[meersdavy]

Although i understand the idea of a special ScenarioRule i am not sure how / where to integrate it in the existing code.

Could you give me some pointers or explain how you think it should work?
Would it make sense to change StepRunner.methodBlock(FrameworkMethod method) and wrap methodInvoker statement with a RunRules statement
( as in BlockJUnit4ClassRunner.methodBlock(FrameworkMethod method) )?

While I am willing to further look into this I am not sure i can free enough time to implement this.