New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
implement --retry #727
Comments
@ericyliu It's coming in cucumber 3. Not sure why Cucumber 2 is still in RC though... it's been out for 2 years |
@charlierudolph or anyone if you had any pointers I would be happy to pick this up and give it a go. My presumptive approach would be that at the end of a scenario running we can tell if it failed or not and re-run it (keeping some state for the retried count). Where can I hook into this? As well as this I would expect at the end of the whole process to be able to access a list of all scenarios, the number of times they retried and their final statuses so this can be a) logged and b) dumped to disk if desired. |
Closing as I personally don't want to implement this as I disagree with it. I don';t like providing a feature that can be used to deal with a flickering test instead of fixing the cause of the flickering. |
👏👏👏 |
@charlierudolph sometimes it's impossible to deal with flickering tests. For example, in our environment we have 23 different services that our application hits, and as we don't own those services any of them can be down at any given time. the --retry flag allows us to rerun the test so a temporary service outage (when they restart the service) doesn't cause our test suite to fail. |
With respect to personal disagreement with this feature (this argument can be as fruitless as tabs vs spaces in the engineering world), if Cucumber as a tooling spec is implementing this we should also support this in the JS version of this tool. It is also in the real world with broad tests suites and complex applications impossible to defend 100% against flakeyness and therefore becomes incredibly expensive. Absolutely you can have an engineering journey to improve it but I doubt there are many companies with the luxury of resources to support such an effort with any decent ROI. In the real world being able to retry failed tests with the addition of good tracking to identify flakeyness for pro-active actioning is super valuable. I'd appreciate if we reconsider the decision to close this issue and as stated before I am more than happy to work on this with some guidance. |
To add to this |
I absolutely hate flakey tests. On my current projects, I am trying to move away from running any flakey test on CI and saving those for manual runs where if something fails due to flakiness you can rerun it or check it manually. This is dependent on finding your source of flakiness and limiting the number of tests that have to deal with it.
Rerun was built so you can easily focus on the tests that need to fixed. Yes, you can use it to retry just the failed scenarios. |
I agree 100% with flaky tests being something to avoid. That being said, the flakiness of a test is often something that's not under a team's control. Here's my situation; I'm using Cucumber, Nightwatch, Selenium and Browserstack to run end-to-end tests on a web app. Often, a scenario will fail due to the inherent flakiness of Selenium, Browserstack or the browser I'm testing in. Interactions involving mouse movements are notorious for this. It's rare for me to be able to run through all of my scenarios without at least a handful being flaky. I can't make Selenium, Browserstack or browsers less flaky, that's just the nature of the tools I have to work with. The solution I need is to be able to retry my scenarios a couple of times if they fail. |
I think we should reconsider this. Cucumber-Ruby has it now. |
Agreed. Here's the code for the Ruby implementation. Could be useful for a JS implementation. https://github.com/cucumber/cucumber-ruby/pull/920/files |
Hi everybody, is the 'retry' feature now available in Cucumber-JS? As much as I agree flaky tests should be fixed, experiences shows it's not always possible when working in large projects with many external dependencies involved. |
Hi @charlierudolph and @aslakhellesoy. Any progress with this feature? |
Nobody is working on it to my knowledge. If anyone provides a pull request I'd consider adding it though. |
Hi @aslakhellesoy thanks for replying so quickly. This is something we're really in need at the moment as one of the devices we need to automate is a bit unstable and in such a case there are no really practical alternatives to just re-running a failed scenario. |
That would be great! Please try to make it behave in the same way as in Cucumber-Ruby. If it diverges from that we may not merge it, as consolidation/consistency is something we're striving for. |
Hi @aslakhellesoy, I've implemented the base retry functionality as the CLI --retry option. I've tested it at work with our project and it works well. I'd need permissions to push my working branch and open a PR to discuss the implementation. Let me know, thanks. |
@hurrikam Great work.. 👏👏👏 |
@charlierudolph could you please help with this? I can't submit my PR at the moment. |
@hurrikam you can create a fork and push your code there then submit a PR from your fork into this repo. https://help.github.com/articles/fork-a-repo/ |
@charlierudolph please see PR-1114 I went through some old conversations in the Cucumber Ruby repo in regard on whether each run of a flaky test should be reported, possibly introducing a 'flaky' result status for the previous runs. Not sure if that was done and again, the draw back would be to get all the attempts of the same test case into the logs. Let's discuss it. |
@hurrikam my team have taken a fork of the PR so we can get this functionality. It's working really well so far. I hope you can get it merged soon as the flag is a huge help! |
@Nick-Lucas That's awesome - Do you have that in? |
@thomaswmanion yep we're actively using the feature. We have a backend that can sometimes get a little choked up flushing through job queues, so it's a been lifesaver. For anyone wanting to use the PR you can point your package.json at my fork (or fork it off me): "dependencies": {
"cucumber": "https://github.com/Nick-Lucas/cucumber-js.git#feature/issue-727-retry",
}, Cucumber is pre-built so it can just be installed into node_modules. It should go without saying that there's definitely no warranty or support as the feature isn't even pre-alpha yet EDIT: I have now left my company and handed over the fork to their github: https://github.com/wonderbill/cucumber-js.git#feature/issue-727-retry |
Nice!!!! Thanks @Nick-Lucas - appreciate the understanding that environments of systems can be flaky in nature! |
This needs to be merged into Master, this isn't closed. |
How can I run the retry feature when running tests? |
@ricardgarcia just use the |
Hi @individual-it , many thanks for the reply. Should it work if I'm using webdriverIO to run my tests on cucumber-js? |
@ricardgarcia As described in #1205, to use this function you can use
as the changes are not yet merged in the main cucumber repo. |
Yes @jain-neeeraj I've used this version in the package.json and it did not work, but as I mentioned I'm running tests using cucumber on webdriverIO, so the following framework "wdio-cucumber-framework". Should it work somehow? How do I run the tests in order to have a retry if I use cucumber-js framework? |
@ricardgarcia lets collaborate at gitter https://gitter.im/cucumber/cucumber-js, I should be able to help you in solving this issue. |
@charlierudolph any chance this could be merged into master? |
it would be nice if there was a feature that retries the test suite X times, and passes a test if that test passes on any of the runs
The text was updated successfully, but these errors were encountered: