New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Result is missing scenario & tags #923
Comments
Nope. You can not easily access the tags of the test case. There is no plan to return to the old scenario result. What is your use case for needing the tags? |
In my case i have dependency on tag system. In order to get some data in the afterhooks. Something like this: @SuiteName @SuiteSectionName <- These tags tell the suite
Feature:
@TC1563697 <- This tag identifies the testcase in the test management tool @New
Scenario:
Given
When
Then |
You can create hooks that run only for scenarios with specific tags: https://github.com/cucumber/cucumber-js/blob/v3.0.1/docs/support_files/hooks.md#tagged-hooks |
I also have several use cases for knowing the scenario tags. Given the following sudo example(missing version syntax a bit as I have test cases in 1.3.1, and im looking at the latest 3.0.1): @set_video @youtube
Scenario: User should see youtube video
@set_video @vimeo
Scenario: User should see vimeo video
this.After({tags: @set_video}, function (testCase) {
let tags = testCase.scenario.tags;
_.forEach(tags, (function (tag) {
if(tag === '@youtube') {
setVideo('youtube');
}
if(tag === '@vimeo') {
setVideo('vimeo');
}
});
} I have one tag that says when the hook should be run and I have other tags that act as data to make the hook more dynamic to work for other use cases. Otherwise I find myself creating the same hook with the same logic and only changing the data I pass it. I find knowing the tags to be very useful and a very powerful tool to making hooks more flexible. Can I also ask here I cant seem to get the result object in the After hook in 3.0.1. I tried testCase, scenarioResult, and scenario. Am I missing something? |
We have a case where we keep our test cases in TestRail and we download them and save as feature files before the run. Then in the after hook we send detailed test results to our SQL database. Those results include:
So with the current change in cucumber 3.x we will never be able to switch as it completely breaks our infrastructure. |
@pawelus My infrastructure does exactly the same thing. Personally I have a wrapper script that launches cucumber. Sounds like you'll have to rearrange your code either way. |
I think losing the ability to know tags in hooks would be a shame. It was the only way to make a little more configurable since you can't pass it parameters I was adding extra tags and then see which were applied to the current scenario to call functions with different inputs. |
@yaronassa we run our tests through Protractor, we do not launch cucumber directly so there's yet another layer of abstraction to count in here. We do download features before Protractor starts as you do but sending results to the database is a different story. With sharding & tests running on Selenium grid and reruns of failed features it will be quite a hassle and complex logic to get the results in the right order in the database. Quite an amount of work to restore the capabilities we had in cucumber 1 and 2. Also creating a custom formatter only to get step results just doesn't sound right. |
Hey, I'm with you. Seeing as cucumberJS is deliberately moving away from this kind of "inner-visibility", I'm offering the kind of solutions I think can work in the foreseeable future. Personally, I think there'll come a time where tinkerers like us would have to resort to overriding inner methods in Cucumber's internal to retain this sort of access. |
If possible i would also back the name of the scenario. Indeed i am adding a snapshot capability and i need tu know the name of the scenario to name my snapshots. |
@gd46 you could do the following: this.After({tags: "@set_video and @youtube"}, function () {
setVideo('youtube');
})
this.After({tags: "@set_video and @vimeo"}, function () {
setVideo('vimeo');
}) That doesn't have any duplication aside from the
Do you need to do this in a You could add screenshots with the |
@charlierudolph I suppose that is a solid replacement solution for my needs. It also removes the need of analyzing the current running scenario. It just might lead to defining several combinations of hooks that account for the different combinations a given function could handle within a hook. So for instance I have similar examples where I get the current running scenarios tags split it based on a pattern and use part of the match as the parameter to a function. In these cases all the steps that need to happen are the same and the only thing that changes is the parameter. So this would cut down on several setup steps in the hook so it could work for multiple cases. One such example: tags: @clear_w2OnlyUser, @clear_w2OnlyArcotEnableUser
I split based on @clear_ and grab the second half as the parameter. tagName coming from the old scenario result of getTags, getName.
let profileToClear = tagName.split('@clear_')[1];
browser.waitForAngularEnabled(false);
browser.get(url);
login();
navigate();
deleteProfile(profileToClear); Let me know your thoughts on this? I do believe your example would be an easy replace in some cases. But in others where there are several extra steps that need to happen you could end up potentially duplicating all these steps. |
My use case is that I setup a mockserver for each scenario, based on the tag I know how it should behave. Adding hooks for specific tagged scenarios is very maintainance-intensive, because I need to add a hook for each scenario that I support... |
We are also using the scenario.name in our Before and After hooks in order to log the name of the scenario. This way we can see when a certain scenario starts and ends when analyzing test results. This change breaks that feature. |
I am using Cucumber-js to drive Selenium. Both local and Browserstack I was using test state (feature, scenario, tags, results, etc) in the hooks for many essential things.
All these were wrapped in apps to enable concurrent instances to reduce overall run time. Why were these dropped? |
Does the scenario being run involve that type of profile? Maybe you could have the scenario save what type of profile is interacting to the world and then you have a single clear tag which prompts the removal of the saved profile? For you mock server could you move the setup from tag based to the using a step that defines what the setup is? Then you can parameterize easily. You can log the scenario line and uri (which is actually unique and prevent you from having to search for the scenario name).
You can use a step for this instead or use unique tags
How were you dynamically skipping tests? #912 adds support for this
You can use the line and uri instead of the name. As noted before, the line and uri are guaranteed unique while the name is not. I also suggest parsing the json formatter or event protocol formatter formatter for test results and using attachments if you need to add anything.
You can use the line + URI
You can use a step for this. From all of these examples, I have yet to a use case that convinces me we should add back tags or name to the hook. Having the tags and name appears to have made certain things easier to do, but I think there are other ways to accomplish them that make more sense to me |
@charlierudolph thanks for the reply. For the mockserver I already had a quick chat with some coworkers and using a shared "background" was in consideration for a solution, so yes you can cross that one of the list. |
@charlierudolph Thanks for the reply. While your suggestions of using Steps and parsing output could be made to work, it’s far inferior to prior versions where callbacks have access to test State.
Oh, and I wasn't actually "Skipping" tests, I was using callback( null, 'pending' ) to avoid running the test. The new 'skip' feature is superior solution. |
@charlierudolph, I was thinking, another solution ... Since the World is available as "this" to all callbacks it would achieve full parity with V2 (provided AfterAll callback were given the Results) |
There is no more scenario object in the system. I don't think parity with v2 should be a goal. I'm glad this conversation came up and am good looking for other solutions but I don't want to go back to what we had previously as I believe that was making users rely much more on the implementation of cucumber and not on its interface. |
Thank you for taking the time to discuss this issue. I truly appreciate the work you've done to provide us with Cucumber-js. But as it stands I cannot upgrade to V3. I don't intend to be combative. I'm just worried and it may come out in my responses. Using only the documented Event interface from the prior version, I have built a framework that dramatically reduces overall test run time by allowing Features to be run concurrently over a huge, configurable test matrix with includes Operating systems/Versions, Browsers/Versions, Desktop/Mobile, and Optimizely A/B tests. I CAN'T use Steps. It's too late. The device type, OS/version and browser/version MUST be determined before the test begins or every Scenario will be forced to have a "set up OS" Step and a "set up Browser" Step. This is in direct conflict with Cucumber's goals. Alternatively, as I mentioned earlier, you could provide the Scenario information (Feature URI, name, line, and Tags) to the World constructor. This is semantically consistent with the intent of the World object and simpler to encapsulate and extend. Not only will this eliminate most of my Event Handlers, but because the Handler's "this" is the World object, it is State consistent with the Observer Pattern.
Having access to the Tags during Before callback allows me to ensure appropriate state based on the browser configuration that is being used to drive the test. Using Tags makes that Declarative. Using Steps obfuscates the Intent The calls, and parameters of the Event handlers in prior versions WERE interface. If devs coupling their code with your implementation leaking out, pass a read-only “Scenario” Proxy to the event handlers. |
@charlierudolph thanks for your reply
True, scenario names are also fragile, but it is a lot less fragile way to reference scenario's than referencing them by file and line. We are using the scenario names in the logging daily to analyze failures in the run and see what went wrong. If the logging only displays the file and line number it will cause way too much overhead to trace back the scenario. Besides that, if someone made changes in the feature file or moved the scenario somewhere else in the meantime this will even get more complicated. |
How is it more overhead? You know the file and line number and can go directly there. If you have the scenario name, you have to search for that string to be taken to that file and line. |
Like I said, if the feature file changed in the meantime and the scenario is not positioned on that line anymore. Especially when I am looking through logs of an older test run (for example 1 week ago). In this case I would have to go through git and see which scenario was on that line at that given time. |
FWIW, cucumber-ruby has two extension points that I think satisfy the principle of "easy things should be easy, hard things should be possible" and might be ideas that could help with these use-cases. First of all, we pass an instance of a Second, we have filters. Filters allow you to manipulate the stream of test cases before they're executed. We use filters to, for example, remove test cases from the stream that don't match the tag pattern specified on the CLI. Filters are a power-user feature. HTH |
I just realised that the documentation behind that link for filters is terrible! Sorry. No time to fix it now, but there are some more examples here: http://www.rubydoc.info/github/cucumber/cucumber-ruby/Cucumber/Filters |
Hi @charlierudolph, Sorry its been awhile since I have follow up on this conversation. Can you elaborate on your statement: "Does the scenario being run involve that type of profile? Maybe you could have the scenario save what type of profile is interacting to the world and then you have a single clear tag which prompts the removal of the saved profile?" Im not sure I follow. And I understand there maybe other ways of obtaining the same solution without the old test result. But from what I have seen so far in this conversation is that all alternative seem more complex then they need to be. The old test result had plenty of descriptive data about the scenario that was about to run and it was the only way to have "parameterized" hooks. Protractor as it stands from what I have seen and tried doesnt currently support running a feature from a scenarios line number. So the new testCase result doesnt have much I find useful other than the status. I would like to see testCase match the the result returned when using getTestCasesFromFileSystem with PickleFilter. I have used this to do some interesting parallel work to support filtering of scenarios by tag to pass to protractor as shards. The information returned from that result is far more useful and I think would be extremely useful to have in testCase result. Example result from PickFilter:
Im not saying it needs to match exactly but I see a lot in this result that others here and I would seem to benefit from. If your interesting in my example, Im using it here: https://github.com/gd46/dibellag-automation-framework/blob/master/configuration.js#L91 |
@charlierudolph Is this where testCase is being set?
From what I can tell there is a reference to pickle along side testCase. So why not return a few more bits from the pickle result onto the testCase result? |
@gd46 okay, lets add |
Hi @charlierudolph, I just saw your comment. That would be great! I'll try to take a look at this soon. I'd be happy to make the contribution. |
@charlierudolph Just wanted to clarify the change. Are we ok with the difference that pickle contains uri without the line number directly on it as sourceLocation does. And if whomever wishes to consume the uri with the line number they can use the line numbers returned from the pickle object? I see no issue with this just wanted to confirm. |
I guess let’s leave the source location object as is (instead of removing it) and add the pickle object. |
Ok that works for me as well. So Im new to contributing to cucumber, Im trying to understand the structure still. As you have pointed out the line that needs to be updated, I believe we can just add something like this next to sourceLocation:
Then anyone who wishes to consume it in the hook can access it as follows:
I have made the updated but Im not 100% sure how to update all the related tests. Would you be able to provide some guidance? |
I was able to test the change by linking my fork with the changes to one of my local projects. The complete testCase result will look like this:
After doing some more testing I noticed that pickle doesnt contain uri at the time we have access to it in test_case_runner. So I think its fine to keep source location as is. This is what PickleFilter returns pickle as:
So everything will be the same minus pickle not having uri in it. |
Opened a PR to highlight the changes so far. Still need to update tests. |
Working on updating tests. I have this thing setup to change my local java version and realized thats why I wasnt able to run some the feature tests :/. |
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Any way to access to scenarioResult.scenario.tags in the new version v3? Is it possible to override somehow test-case-finished event to pass old model scenario_result? Thanks
The text was updated successfully, but these errors were encountered: