scenario_autoretry does not reset runner.hook_failures

[rittneje]

The scenario_autoretry monkey patch does not intervene with or reset the increment of self.hook_failures in the runner. Consequently, if before_scenario/after_scenario fails the first time, and then passes on the retry, behave still exits with failure at the end because hook_failures is non-zero.

behave/behave/runner.py

Line 649 in 1c6197b

self.hook_failures += 1

behave/behave/runner.py

Lines 751 to 755 in 1c6197b

	failed = ((failed_count > 0) or self.aborted or (self.hook_failures > 0)
	or (len(self.undefined_steps) > undefined_steps_initial_size)
	or cleanups_failed)
	# XXX-MAYBE: or context.failed)
	return failed

[jenisys]

Mmh, you seem to have flaky tests.
Therefore, best practice is to fix the flaky tests.

The scenario_autoretry is proof-of-concept code / best-effort code.
Come up with a better solution.

SOLUTION SKETCH:
You probably need to record the test-runner state before the scenario is run to be able to restore this state if you retry a scenario. If the test-runner state currently contains only the hooks_failed count, the solution will become even simpler.

SEE ALSO:

• behave/contrib/scenario_autoretry.py

[rittneje]

@jenisys Yes obviously we'd prefer the tests not to be flaky. But reality ensues.

I don't know what you mean by the "test-runner state".

[jenisys]

Test-runner state (in this case) is any group of variables in the test-runner that you need to restore when you retry a scenario. As already mentioned, this test-runner state contains at least runner.hooks_failed counts (and probably only that variable).