Make JSON output to be ready to convert to JUnitXML for CI usage

[ghost]

Before:
When setting "resultJsonOutputFile" configuration to save json output, the format is not ready for CI.

After:
The output file will be ready for converting to JUnitXML, which is a common format consumed by CI.

[googlebot]

Thanks for your pull request.

It looks like this may be your first contribution to a Google open source project, in which case you'll need to sign a Contributor License Agreement (CLA) at https://cla.developers.google.com/.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check the information on your CLA or see this help article on setting the email on your git commits.

Once you've done that, please reply here to let us know. If you signed the CLA as a corporation, please let us know the company's name.

[ghost]

Just signed CLA. Thanks.

[googlebot]

CLAs look good, thanks!

[hankduan]

The purpose of the 'resultJsonOutputFile' reporter isn't for it to match a cucumber log. The purpose is for jasmine/mocha/cucumber to be able to share a common output. If you just want to report specific items for cucumber, you can pass your reporter into cucumber as a custom reporter.

[ghost]

One question about 'resultJsonOutputFile'. What's the purpose for this jasmine/mocha/cucumber common output? We thought it could also be for CI, so we tried the approach to make it easier to convert to JUnitXML format. And this common output can be more functional.

Can you spare some time to check on the output result? It's more structural with more test details.

Thanks for your time.

[ghost]

Changed the PR title to meet the purpose of CI usage.

[jlin412]

@sonyschan This sounds like a very helpful PR. Do you think the problem is coming from json formatter in cucumber.js? Maybe the PR should be in cucumber.js instead?

[ghost]

@jlin412 cucumber.js is already using such json formatter to log Cucumber processes, and quite complete implemented Cucumber concept. So it looks like not really a "problem", just need someone to save the log/result into a well formatted json file, which can be easily put into CI to generate beautiful reports.

And Protractor is exact the one who already implemented "resultJsonOutputFile" configuration and functionality. If the Json output file is already there, why not improve it to be more CI friendly?

However we are not sure why jasmine/mocha/cucumber these three framework in Protractor need to be shared as common output. Using different testing framework to test against one project looks like not really common cases. Maybe there are some reasons for this?

With the Json format being improved, we don't need report generator like jasmine-reporters which require developers to pay attention to the configuration and match the correct reporter versions. All we need is a parser to convert .json to .xml.(And there are already parsers for this)

[jlin412]

@sonyschan are you saying that the problem is using resultJsonOutputFile? If you try to generate json format from cucumber.js directly, does it work in Jenkins CI? Currently, I have no issue using output json from protractor/cucumber with Cucumber Plugin in Jenkins CI. Also, I haven't tested resultJsonOutputFile feature, does it require setting below?

  cucumberOpts: {
    format: 'json'
  }

[hankduan]

As an end user who only use 1 framework, being able to combine the three frameworks might not be important to you, but as a framework developer, it is very important because it allows us (including people looking to contribute) to introduce a single feature to all protractor frameworks without going to each individual framework and learning how to implement a feature for it. This results in easier learning curve, faster feature delivery and easier maintainability.

For example, we use this to write tests to test protractor -- i.e. asserting stacktrace is sane, error code is correct, right errors are being thrown, etc. Without a common reporter, we would need to go to adapt this backend test framework for each of jasmine/mocha/cucumber.

That being said, developers are more than welcome to add more useful output to resultJsonOutputFile if it is useful or modify the current output if it doesn't make sense. However, for your case, implementing new reporters for test frameworks (i.e. jasmine/mocha/cucumber) is not the responsibility of protractor. What if tomorrow another person comes and wants resultJsonOutputFile to report a different output that breaks your conversion to .xml, but say helps conversion to .html? That's why protractor is built so that people can easily attach other reporter modules to it, so that everyone can use the most appropriate report for their use case, and not try to implement all reporters itself.

[juliemr]

Agreed, the primary purpose of the JSON output is to serve as a common language between test frameworks. This is for Protractors own use as well as plugins. The documentation for this JSON is in https://github.com/angular/protractor/blob/master/lib/frameworks/README.md. I'm going close this PR since it will break things for us by updating only cucumber.

We could take a look at a another change which edits the spec result format for all frameworks and updates the associated documentation and usage from plugins. Note that this would probably introduce a breaking change, so we'd need to pick a good time to release. If we do it, I'd like it to be so that the output matches a format agreed upon by a group such as that being discussed at https://github.com/js-reporters/js-reporters

[ghost]

@jlin412 Here is the resultJsonOutputFile https://github.com/angular/protractor/blob/master/docs/referenceConf.js#L229

Originally I would like to keep cucumberOpts format as 'pretty' for dev environment, and use resultJsonOutputFile for CI usage. Now I know resultJsonOutputFile is for framework developing usage, and is designed to be as general as possible.

Can you tell me how to generate json from cucumberjs directly? I am curious about this. Or I may need to create a custom reporter for my use case.

@hankduan & @juliemr
Thanks for your comprehensive explanations. I'll find a better way to general reports.

[hankduan]

Googling "cucumber reporter" brings up a bunch. In terms of which one is good though, I have no idea (sorry) as my only exposure with cucumber is to implement resultJsonOutputFile =)
You can ask your question on stackoverflow; it's a much better medium for this type of questions

[jlin412]

@sonyschan Take a look of cucumber/cucumber-js#90
I have two approaches listed there. Currently, I use the second approach with a hook so I can have both pretty and json output together.