Screenshots seemed not to be sent using the afterStep hook

[gromanas]

The problem

I am trying to use the cucumber example to attach images into report portal and the image does not post.

Environment

• WebdriverIO version: 5.12.5
• wdio-reportportal-version version: 5.0.1
• Node.js version: 10.15.1
• WDIO Testrunner and version(cucumber): 5.12.3
• Additional wdio packages used (if applicable):

    "@babel/core": "7.5.5",
    "@babel/plugin-proposal-decorators": "7.4.4",
    "@babel/plugin-transform-runtime": "7.5.5",
    "@babel/preset-env": "7.5.5",
    "@babel/register": "7.5.5",
    "@babel/runtime": "7.5.5",
    "@lingui/cli": "2.8.3",
    "@react-native-community/eslint-config": "0.0.5",
    "@testing-library/react-hooks": "2.0.1",
    "@wdio/appium-service": "5.12.1",
    "@wdio/cli": "5.12.5",
    "@wdio/cucumber-framework": "5.12.3",
    "@wdio/dot-reporter": "5.12.1",
    "@wdio/local-runner": "5.12.5",
    "@wdio/sauce-service": "5.12.5",
    "@wdio/spec-reporter": "5.12.1",
    "@wdio/sync": "5.12.3",
    "@welldone-software/why-did-you-render": "3.3.3",
    "appium": "1.14.0",
    "babel-core": "7.0.0-bridge.0",
    "babel-eslint": "10.0.3",
    "babel-jest": "24.9.0",
    "babel-plugin-functional-hmr": "1.0.25",
    "babel-plugin-macros": "2.6.1",
    "babel-plugin-module-resolver": "3.2.0",
    "babel-plugin-transform-remove-console": "6.9.4",
    "body-parser": "1.19.0",
    "chai": "4.2.0",
    "chai-as-promised": "7.1.1",
    "chance": "1.1.0",
    "enzyme-to-json": "3.4.0",
    "eslint": "6.3.0",
    "eslint-config-airbnb": "18.0.1",
    "eslint-plugin-flowtype": "4.2.0",
    "eslint-plugin-import": "2.18.2",
    "eslint-plugin-jsx-a11y": "6.2.3",
    "eslint-plugin-prettier": "3.1.0",
    "eslint-plugin-react": "7.14.3",
    "eslint-plugin-react-hooks": "2.0.1",
    "eslint-plugin-react-native": "3.7.0",
    "eslint-plugin-react-native-a11y": "1.2.0",
    "flow-bin": "0.106.3",
    "fs-extra": "8.1.0",
    "husky": "3.0.5",
    "jest": "24.9.0",
    "jest-extended": "0.11.2",
    "jest-styled-components": "6.3.3",
    "jetifier": "1.6.4",
    "live-server": "1.2.1",
    "metro-react-native-babel-preset": "0.56.0",
    "metro-with-symlinks": "1.3.2",
    "minimist": "1.2.0",
    "multiple-cucumber-html-reporter": "1.13.0",
    "ng-apimock": "1.4.9",
    "prettier-eslint": "9.0.0",
    "react-test-renderer": "16.8.6",
    "reactotron-react-native": "3.6.5",
    "reactotron-redux": "3.1.1",
    "wdio-cucumber-snippet-reporter": "0.0.5",
    "wdio-cucumberjs-json-reporter": "1.0.8",
    "wdio-native-app-compare-service": "1.1.0",
    "wdio-reportportal-reporter": "5.0.1",
    "wdio-reportportal-service": "5.0.2",
    "yarn-check": "0.0.3"
  },  

Details

Hello all! My question is that I am trying to integrate rp to my project. I have successfully install rp but I am facing an issue on how to send a screenshot to rp. I was trying to use the example in ReadMe file. According to https://webdriver.io/docs/wdio-reportportal-reporter.html I need to add it in afterStep. What I have seen the hook has been change from afterStep: function (uri, feature, scenario, step, result) to afterStep: function (uri, feature, { error, result }).

Also some logs from sendToReporter() that is used from sendFileToTest()

 I am in sendToReporter ------------------->>>>>>>>>>>>> rp:failedFile { test: 'failureObject',
  level: 'ERROR',
  name: 'screenshot.png',
  content: <Buffer 89 50 4e 47 0d 0a 1a 0a 00 00 00 0d 49 48 44 52 00 00 04 65 00 00 09 84 08 02 00 00 00 23 47 44 ee 00 00 00 01 73 52 47 42 00 ae ce 1c e9 00 00 00 1c ... >,
  type: 'image/png' }

Code To Reproduce Issue [ Good To Have ]

 afterStep: function (uri, feature, { error }) {
    if (error.name === 'Error') {
      const screenShot = global.browser.takeScreenshot();
      let attachment = Buffer.from(screenShot, 'base64');
      reportportal.sendFileToTest('failureObject', 'ERROR', "screenshot.png", attachment);
    }
  },

[BorisOsipov]

Submit an issue to wdio repo to make after step hook compatible(at least provide for which step hooks is executing). You have to know which step failed to send screenshot to it. I am not sure that something should be done in reporter code.

[jpaolini]

I came across the same issue when using webdriverio v5 and ReportPortal v5. The afterStep hook signature was changed when it was updated so it respects promises and runs synchronously. You can see the issue here.

Note: I've noticed in the afterStep (From the docs here.)

afterStep: function (uri, feature, { error, result }) {
},

That the result is always undefined, so I wasn't able to use this function.

Instead the first approach I tried was to set the autoAttachScreenshots to true and then in the afterScenario do the following:

afterScenario: function (uri, feature, scenario, result, sourceLocation) {
  if (result.status === 'failed') {
    browser.takeScreenshot();
  }
},

This won't work since the wdio-reportportal-reporter receives the onTestFail event before the afterScenario which removes the test item and when it attempts to attach the screenshot, it won't find the test to attach it too. I started looking into this and believe I have a solution however, I'm currently short on time.

The primary issue comes down to being able to get the step that ran so that we can attach the screenshot to it. As a workaround (I'm not a fan of the approach) to get things working I created a custom reporter and required it in my wdio.conf.js file. Below is the reporter:

const WDIOReporter = require('@wdio/reporter').default;
const reportportal = require('wdio-reportportal-reporter');

export default class ReportPortalReporter extends WDIOReporter {
  async onTestFail(test) {
    const { title, fullTitle } = test;

    const screenshot = await browser.takeScreenshot();
    const attachment = Buffer.from(screenshot, 'base64');
    reportportal.sendFileToTest({ title }, 'info', 'screenshot.png', attachment);
  }
}

This was the only way I was able to get the step so that I could attach the screenshot correctly. Another thing to note, when using Report Portal v5, that if you use log level error instead of into then the auto-analysis won't work, it'll throw a parsing error.

This isn't the best solution, but for me it's not to invasive and suites our needs while we wait for Report Portal v5 to come out of beta.

[BorisOsipov]

@jpaolini thanks for sharing your thoughts. I've made some quick research and came to the same conclusions.

As a workaround (I'm not a fan of the approach)

yeah. it isn't good to use browser in reporter..

I'm currently short on time. Hope will make solution for this and implement better Cucumber steps support for Report Portal V5 by the end of September.

[gromanas]

@jpaolini Indeed I was able to see that screenshot with the workaround you suggest. I had the same issue so I think we should open also an issue for the undefined in result on the afterStep: function (uri, feature, { error, result }) { },.

[BorisOsipov]

I think we should open also an issue for the undefined in result

@gromanas yes. Please, submit the issue, simple repo to reproduce will help us in webdriverio repo fix\research it.

[jpaolini]

@BorisOsipov

yeah. it isn't good to use browser in reporter..

It definitely feels dirty, lol

I was thinking of looking into how to leverage the autoAttachScreenshots and see if there was a way to prevent the condition where the testFinished runs before the afterScenario so we could just auto attach the screenshot. Maybe a different way to manage the storage mechanism around getting the current test. What are your thoughts? I think I can find some time to help with the work before the month end.

[BorisOsipov]

way to prevent the condition where the testFinished runs before the afterScenario so we could just auto attach the screenshot. Maybe a different way to manage the storage mechanism around getting the current test.

1. After failed step event we immediately will have bunch of skipped test events for rest of scenario steps. So currentTest() will point to last skipped step any way when afterScenario occurs.
2. sorry, I am not happy changing getting current test - it looks like crutch. currenttest() uses for making correct test items hierarchy for reportportal.

My thoughts

1. when i looked into this I couldn't find a way how to get current cucumber step name in wdio\cucumber from hooks - it is a weird issue - reporter.sendLogToTest \ reporter.sendFileToTest
   can no longer be used for Cucumber at all. But seems scenario name it still available in wdio afterStep hook.
2. So, I think we can do the following:

1. Add status \ scenarioName fields to StorageEntity and update status(failed\passed\skipped) in testFinished()(I will do it any way to support nested steps in reportportal v5)
2. Add special methods for cucumber like send(File|Log)ToFailedStepInScenario(scenarioname: String, ...) and in these methods find corresponding failed step from getStartedTests() list for scenarioname to attach log\file.

I think it is the best that we can do right now. Another approach is fix wdio-cucumber adapter to pass step name to afterStep hook, but I am not ready to do it.

What do you think? @jpaolini

[gromanas]

I have opened the issue for the afterStep hook in wdio repo
webdriverio/webdriverio#4489

[jpaolini]

1. After failed step event we immediately will have bunch of skipped test events for rest of scenario steps. So currentTest() will point to last skipped step any way when afterScenario occurs.
2. sorry, I am not happy changing getting current test - it looks like crutch. currenttest() uses for making correct test items hierarchy for reportportal.

Those are good points, I forgot about the skipped test events. Agree, that changing the current test is more of a work around.

My thoughts

1. when i looked into this I couldn't find a way how to get current cucumber step name in wdio\cucumber from hooks - it is a weird issue - reporter.sendLogToTest \ reporter.sendFileToTest
   can no longer be used for Cucumber at all. But seems scenario name it still available in wdio afterStep hook.

2. So, I think we can do the following:

3. Add status \ scenarioName fields to StorageEntity and update status(failed\passed\skipped) in testFinished()(I will do it any way to support nested steps in reportportal v5)

4. Add special methods for cucumber like send(File|Log)ToFailedStepInScenario(scenarioname: String, ...) and in these methods find corresponding failed step from getStartedTests() list for scenarioname to attach log\file.

I think it is the best that we can do right now. Another approach is fix wdio-cucumber adapter to pass step name to afterStep hook, but I am not ready to do it.

What do you think? @jpaolini

I think that would work nicely and be a relatively quick win to help with some of the issues folks are having.

For the long term though, I think the real win would be to go to the source of all of the headache which is cucumberjs not providing a before/after step hook. If that get implemented then it'll funnel through the stack and we wouldn't need the work arounds. I was looking at this pr and it seems like the updates to have the after/before steps respect the rules (promises/synchronous execution) of the other wdio hooks is the reason why it doesn't have the step definition anymore. I think it's worth investigating cucumberjs to see the work involved to add those hooks as part of this.

[mgrybyk]

Closing this one as far as it is reportportal related.

[BorisOsipov]

@mgrybyk Need your advice as wdio-cucumber adapter expert. Are there any way to pass cucumber step name to wdio afterStep hook as parameter?(it was deleted in https://github.com/webdriverio/webdriverio/pull/4288/files) I didn't find how to change current implementation to achieve it - in Cucumber.setDefinitionFunctionWrapper we know nothing about step name(

[mgrybyk]

There is no easy way to do it, it's not something like pass an additional argument.
I'm thinking on adding before step listener, storing current step name and then reuse it in after step hook.
Are you going to start working on it right now? If no I can try help you next week because I want to finish my PR first

[BorisOsipov]

I am not familiar with cucumber and v5 cucumber integration. If you can help us with we really appreciate it and can wait necessary time

[BorisOsipov]

Now working well. Check readme for implementation details.

@mgrybyk many thanks for help with wdio-cucumber enhancements