MERGE_FEATURES_WITH_RETEST does not update the job status

[lemitrou]

I have build a test framework with Java + cucumber + junit.
The tests are run with surefire plugin, and below is the 2 runners :

• AllTestRunner.java that runs all the test
• FailTestRunner.java that run the fail test of the 1st run

I generate a cucumber report - using 5.6.1, and in plugin configuration i've this :
<mergeFeaturesWithRetest>true</mergeFeaturesWithRetest>

Problem : The jenkins job mistakenly reported as Failure when tests at rerun are pass.
Here are some meaningful stdout logs :
surefire logs :

Failed tests: 
Sceanrio
  Run 1: expected:<2> but was:<1>
  Run 2: PASS

which means that test failed in first run and passed in re run

and cucumber report marks the job as FAILURE

var/jenkins_home/jobs/e2e-tests-dev/builds/858/cucumber-html-reports/.cache/pf-e2e-bdd/target/report/cucumber.json
[CucumberReport] /var/jenkins_home/jobs/e2e-tests-dev/builds/858/cucumber-html-reports/.cache/pf-e2e-bdd/target/report/cucumber_rerun.json
[CucumberReport] Found 11.111111 failed steps, while expected not more than 0.000000 percent
[CucumberReport] Build status is changed to 
Build step 'Cucumber reports' changed build result to FAILURE
Performing Post build task...

As documentation reports for mergeFeaturesWithRetest:
merge features and scenarios from different JSON files of different runs into a single report by features

Since all json files with the results, merged to one file, and elements with same ids overwritten with the newer, then
I believe that make sense to run all the checks for fail steps ,scenarios, features e.t.c against final json , and not to all initial files, what do you think???

[damianszczepanik]

Truly I have no experiences with this flag. It was implemented and tested by one contributor

[udayRepository]

Hi @lemitrou any work around on this issue? i do face the same issue, let me know on this , much appreciated.

Thanks
Uday

[lemitrou]

Hi @udayRepository I don't work anymore on the project. Fyi

[tarasmytlovych]

Hi I am also interested in some solution or workaround