[Feature] Track Code Coverage via istanbul

[mxschmitt]

Update: It can be easily integrated in the user-land now.

For that follow the guide here.

For any suggestions please comment inside this issue.

[barakbs1]

We currently use jest-playwrights' code coverage feature and we depend on it. We would love to migrate to new playwright test-runner but we have to have the code coverage feature, and would really hope it would be integrated.
Also it would be amazing if there would be an integration with Kibana-Elastisearch for displaying the results. Moreover, we currently use this % test coverage as a bitbucket-pipeline gate, so an integration with bitbucket pipelines would be amazing!

Keep up the amazing work guys! 🚀 🥇

[mpuertao]

Thanks, i need coverage with istanbul. stay tuned

[mxschmitt]

Small update, we created an example guide how you can use it. It is currently there to get an understanding which additional features are necessary to provide a full story around code coverage collections. For any suggestions / feedback it would be highly appreciated to get input especially:

• If you are using the babel-plugin-istanbul?
• If single browser code coverage like Chromium would be enough for you?
• If you would prefer an approach with minified JS + source maps instead of using the babel plugin? (this would require to have the source maps either locally or available on the web-server)
• What are you doing with the coverage data? text-summary, uploading it to a provider, serving the html, only for manual checking etc.

[mpuertao]

For me:

• If you are using the babel-plugin-istanbul? R:// Yes.
• If single browser code coverage like Chromium would be enough for you? R:// Yes, the coverage is a quantitative aspect, and with a single browser I can determine what components I am covering from the E2E tests.
• What are you doing with the coverage data? text-summary, uploading it to a provider, serving the html, only for manual checking etc. R:// Generate lcov.info o json o html, we are send this data to elasticsearch and after grafana

[anfruiz]

I implemented the baseFixtures.ts in the test folder and I created my test like this:

import { test } from "./baseFixtures";
test('P', async ({ page }) => {
  await page.goto("http://localhost:3000");
  const commandLine = await page.innerText("p");
  test.expect(commandLine).toBe("Edit src/App.js and save to reload.");
});

[mxschmitt]

How did you add the babel-plugin? This requires a transcompilation step which is not out-of-the box available. This looks like create-react-app and needs by that an ejected installation.

[anfruiz]

This requires a transcompilation step which is not out-of-the box available

I only created the .babelrc file in the root project. That's means that I can't apply coverage if I use creat-react-app or ng new my-app?

[mxschmitt]

create-react-app does not use the babelrc file. For that you would need to either eject your create-react-app or use react-app-rewrired

[anfruiz]

Ok thanks Max, for an angular app is the same case or I can create a babelrc file?

[mxschmitt]

I'm not familiar with Angular, depends how/if they allow you to add a custom babel plugin.

[anfruiz]

Hi Max, when I create the baseFixtures.ts in an angular app, it doesn't recognize these 3 dependencies but if I create it in other project the dependencies are ok

[mxschmitt]

hi, this is only a typings warning. It should still be able to work correctly. To get rid of the typing warnings, replacing it with

import * as fs from 'fs';
import * as path from 'path';
import * as crypto from 'crypto';

should fix. Would be awesome if you could verify, then I can adjust it properly.

[anfruiz]

Excellent, this solution fix the warnings!! Thanks!!

[dmail]

According to my tests Istanbul coverage and v8 coverage cannot be merged together. I have opened an issue to track this: istanbuljs/v8-to-istanbul#144. Maybe there is a way but I couldn't make it work.

If confirmed it means coverage collected using Istanbul cannot be merged with coverage returned by page.coverage.stopJSCoverage(). Dropping this here in case you was not aware of this issue.

[scottfwalter]

Excellent, this solution fix the warnings!! Thanks!!

Do you have a sample repo that shows integration with Angular?

[mxschmitt]

Closing since its possible in the user-land, see https://github.com/mxschmitt/playwright-test-coverage

[anishkny]

@mxschmitt It would be great if this were offered as part of Playwright itself via an option instead of in user-land. Thoughts?

[CodingProgrammer]

@mxschmitt Hi mx, this coverage tool can only work within the business project ? Can I separate the e2e automation test from business code?

[gauravgandhi1315]

Update: It can be easily integrated in the user-land now.

For that follow the guide here.

For any suggestions please comment inside this issue.

@mxschmitt Do you have baseFixture.ts in js? My framework in JavaScript bases

[dhythm]

Does anyone collect the coverage of playwright component testing with babel-plugin-istanbul?

It doesn't collect the coverage in my local using vite + vite-plugin-istanbul + playwright-ct.
(FYI: the normal playwright works well 👍 )
As mentioned in #14511, it would help us if baseFixture.ts supports playwright component testing.

I'm not sure but the following code should be updated because playwright-ct uses mount that wraps page instead of page directly.

[dhythm]

Does anyone collect the coverage of playwright component testing with babel-plugin-istanbul?

It doesn't collect the coverage in my local using vite + vite-plugin-istanbul + playwright-ct. (FYI: the normal playwright works well 👍 ) As mentioned in #14511, it would help us if baseFixture.ts supports playwright component testing.

I'm not sure but the following code should be updated because playwright-ct uses mount that wraps page instead of page directly.

Sorry, the above comment was my failure 🤦‍♂️
baseFixture.ts works well for component-testing too.

[jeffhuys]

Sorry, the above comment was my failure 🤦‍♂️ baseFixture.ts works well for component-testing too.

Hey @dhythm, I'm trying to get the same combination working, but it doesn't even work with normal playwright.

I also have vite + vite-plugin-istanbul + playwright(-ct).

In my vite.config.ts, I have:

istanbul({
  include: 'src/*',
  exclude: ['node_modules', 'tests/', 'src/tests/'],
  extension: ['.js', '.ts', '.vue'],
  requireEnv: true,
})

I have the same code as your screenshot, where did you put it? beforeEach and afterEach don't seem to work, and the __coverage__ seems to be undefined (as well as collectIstanbulCoverage()).

While I'm figuring this out, I'm running my tests using npx nyc playwright test tests/integration/login.spec.ts.

Anything I'm missing here?

EDIT: I fixed NORMAL playwright by changing the command I'm running to npx nyc --all --cwd . --include "src/*" npm run test:integration (and then report using npx nyc report --cwd . --reporter=html).

However, when I run the same, but with test:component, I get: page.evaluate: ReferenceError: cov_2is1g4bs0e is not defined. If someone knows why this occurs, please let me know!

    page.evaluate: ReferenceError: cov_2is1g4bs0e is not defined

        at eval (eval at evaluate (:197:30), <anonymous>:1:6)
        at UtilityScript.evaluate (<anonymous>:199:17)
        at UtilityScript.<anonymous> (<anonymous>:1:44)
        at Object.context [as fn] (/Users/jeff/Development/redacted/src/tests/baseFixtures.ts:2:4523)

[gselsidi]

is there an option in playwright config to turn this on?

[bradyisom]

Does anyone collect the coverage of playwright component testing with babel-plugin-istanbul?
It doesn't collect the coverage in my local using vite + vite-plugin-istanbul + playwright-ct. (FYI: the normal playwright works well 👍 ) As mentioned in #14511, it would help us if baseFixture.ts supports playwright component testing.
I'm not sure but the following code should be updated because playwright-ct uses mount that wraps page instead of page directly.

Sorry, the above comment was my failure 🤦‍♂️ baseFixture.ts works well for component-testing too.

@dhythm, I am in the same position you were. I have my normal Playwright tests working with coverage using vite, vite-plugin-istanbul and baseFixture.ts. However, coverage is not getting collected with the same setup and playwright-ct. Can you please share what your problem was and how you got it working for component testing?

[prests]

@bradyisom do you have an example of how you were able to get normal playwright tests working with vite-plugin-istanbul? I am using vite with vue3 and the coverage report doesn't include any .vue files. Regardless of if you're using vue or not I'd love to see a setup where vite-plugin-stanbul is working with playwright.

I'm not sure why this isn't a feature within playwright to check for instrumented code. Other integration packages like cypress offer this functionality with low effort from the consuming app. Would love to see the same here!

[bradyisom]

@prests, I don't have a simple repro, but there were a few key pieces for me to get it working:

1. Add the vite-plugin-istanbul plugin to vite.config.js
2. Add baseFixtures.js See here and use the test and expect exported from there. You can also use the playwright-test-coverage NPM module, as it does pretty much the same thing.
3. Output for the coverage ends up in the .nyc_output directory after running tests like this. To generate a report, run nyc report after the tests run.

[iamdhruv]

Does anyone collect the coverage of playwright component testing with babel-plugin-istanbul?
It doesn't collect the coverage in my local using vite + vite-plugin-istanbul + playwright-ct. (FYI: the normal playwright works well 👍 ) As mentioned in #14511, it would help us if baseFixture.ts supports playwright component testing.
I'm not sure but the following code should be updated because playwright-ct uses mount that wraps page instead of page directly.

Sorry, the above comment was my failure 🤦‍♂️ baseFixture.ts works well for component-testing too.

Hey @dhythm, will you be able to share an example repo with your changes, we are struggling to make this work

[cenfun]

Just integrate the code coverage report into monocart-reporter
There are 2 supported data inputs:

• V8
• istanbul

If you are interested, try it according to the examples:
1, https://github.com/cenfun/monocart-reporter-test/tree/main/tests/coverage
2, https://github.com/cenfun/monocart-reporter/tree/main/tests/report-coverage

[edumserrano]

TLDR: see this comment for detailed guidance and code demo on setting up code coverage.

@mxschmitt This issue was opened a couple of years ago and I was wondering if you know what the plans are for this? Meaning, do you plan to provide a better "out of the box" experience to get code coverage?

Thank you for your guide to get code coverage with istanbul. Thanks to it I've been able to get code coverage working in my angular app which is started by playwright's webServer before the tests run.

However, my journey to get code coverage working in Playwright was very painful. I suspect I'm not alone on this. I think a better out of the box experience or at least better docs outlining different ways to get code coverage to work would be a welcome improvement.

I believe @anishkny is right when he says this would be better if it wasn't something only in user-land. 🙏

My Playwright code coverage journey

I started by looking at the official docs on class-coverage and I couldn't get the example in the docs to work for my scenario. My test was just this:

test('basic test', async ({ page }) => {
  if ([test.info](http://test.info/)().project.name === 'chromium') {
    await page.coverage.startJSCoverage();
  }
  await page.goto('/');
  if ([test.info](http://test.info/)().project.name === 'chromium') {
    const jsCoverage = await page.coverage.stopJSCoverage();
    for (const entry of jsCoverage) {
      const converter = v8toIstanbul('', 0, {
        source: entry.source,
      });
      await converter.load();
      converter.applyCoverage(entry.functions);
      const coverageJSON = JSON.stringify(converter.toIstanbul());
    }
  }
});

And my playwright.config.ts had the following:

use: {
  baseURL: "http://127.0.0.1:4999"
},
webServer: {
  command: "npx ng serve --port 4999",
  url: "http://127.0.0.1:4999",
  reuseExistingServer: !_env.CI
}

In terms of the test itself, all is fine, playwright's webServer starts the angular app with ng serve and then is able to navigate to /. The problem is that the line await converter.load(); fails with:

Error: An error occurred while trying to read the map file at c:\dev\repos\temp\my-app\runtime.js.map
Error: ENOENT: no such file or directory, open 'c:\dev\repos\temp\my-app\runtime.js.map'
at readFromFileMap (c:\dev\repos\temp\my-app\node_modules\convert-source-map\index.js:60:11)
at Converter (c:\dev\repos\temp\my-app\node_modules\convert-source-map\index.js:67:32)
at Object.exports.fromMapFileComment (c:\dev\repos\temp\my-app\node_modules\convert-source-map\index.js:153:10)
at Object.exports.fromMapFileSource (c:\dev\repos\temp\my-app\node_modules\convert-source-map\index.js:165:22)
at V8ToIstanbul.load (c:\dev\repos\temp\my-app\node_modules\v8-to-istanbul\lib\v8-to-istanbul.js:52:66)
at tests\example.spec.ts:33:23

@cenfun was kind to help me on Playwright's discord channel by answering on my help post Help with code coverage example from the docs. He pointed out that the problem was that v8toIstanbul requires the right source path and linked to [BUG] Code coverage failing with reading map file.

At this point I decided to pivot, I'm not an experienced frontend developer and I didn't feel comfortable trying to do what @cenfun suggested to get the v8toIstanbul working.

I then found this issue and adapted your guide to my Angular project. This again wasn't a straightforward thing for me to do. I had to learn a bit about how Angular doesn't expose it's webpack configuration and possible ways to extend it. I eventually figured it out and was able to extend my Angular app so that it got instrumented via the babel-plugin-istanbul.

At this point I was happy because I got code coverage to work. However, it felt that I had to do too much:

1. install babel-plugin-istanbul
2. update my angular.json to use a different builder so that I could extend Angular's webpack configuration. I decided to use ngx-build-plus for this.
3. create a base fixture with some code that is responsible for recording istanbul's code coverage as per your example. I later found out I didn't need to create this base fixture myself, I could use the Playwright Test Coverage package to do the same.
4. make sure my tests imported the test and expect from the base fixture instead of doing import { expect, test } from "@playwright/test";

The above felt a bit much and it also felt like something I'd have to document well on my project's README because my team/other people in the company would not easily grasp all the moving parts.

Feeling that the above was too much I decided to pivot again and try @cenfun 's monocart-reporter. I eventually found Playwright Component Testing Vue Example and following the instructions on that repo I was able to get monocart-reporter to produce code coverage for my app.

Unfortunately the code coverage from V8 + monocart-reporter wasn't showing accurately. Talking further with @cenfun , he recommended I create an issue for this which can be followed here.

The end of my journey

The whole journey took about 30 hours to complete. I started thinking "this should be simple" and proceeded to hit head-first into a wall of errors and questions of "nothing works".

Now I understand much more about code coverage in general in javascript projects, not just regarding playwright and if I had a lot of this base code coverage knowledge when I started the journey I feel like I could have got things to work much quicker.

However, the point still remains that, at least for developers less experienced with frontend apps, it's way more difficult to get code coverage working than it should be and requires learning/knowing about several concepts. Furthermore, the example in the docs just didn't work at all for my scenario which used the webServer to start my angular app.

Related issues and links

• Throws when source map is not found istanbuljs/v8-to-istanbul#103
• [BUG] Issue with coverage after istanbul conversion #7109
• [BUG] Code coverage failing with reading map file #21626
• [Feature] support test coverage for component testing #14511
• Add ability to calculate code coverage for component tests #22953
• [BUG] Inaccurate code coverage cenfun/monocart-reporter#29
• [BUG] Incorrect code coverage when converting from V8 to istanbul cenfun/monocart-reporter#50
• Use Istanbul coverage collection with Playwright Test
• Monocart reporter
• Playwright Component Testing Vue Example
• Monocart code coverage demo

[scottfwalter]

@edumserrano can you share your webpack config to get babel to work with Angular?

[edumserrano]

@scottfwalter see the link Monocart code coverage demo from the Related issues and links on my previous comment.

That repo contains an Angular app and is setup to allow both type of code coverage instrumentations:

• using babel+istanbul: see https://github.com/edumserrano/monocart-code-coverage-demo/tree/main/tests/babel-istanbul
• using V8+monocart-reporter: see https://github.com/edumserrano/monocart-code-coverage-demo/tree/main/tests/monocart

As per the README, the way the demo repo is setup is that when you run npm run test-istanbul it will use the angular.json file from https://github.com/edumserrano/monocart-code-coverage-demo/blob/main/angular-configs/angular-babel-istanbul.json.

When you run npm run test-monocart it will use the angular.json file from https://github.com/edumserrano/monocart-code-coverage-demo/blob/main/angular-configs/angular-monocart.json.

The webpack part you're looking for is done by using ngx-build-plus to extend Angular's webpack config. See https://github.com/edumserrano/monocart-code-coverage-demo/blob/4b58a16fcf224a59db421a788baedc77f6b69c20/angular-configs/angular-babel-istanbul.json#LL85C5-L85C5 and https://github.com/edumserrano/monocart-code-coverage-demo/blob/main/coverage.webpack.js.

[scottfwalter]

@edumserrano Thanks! I asked because we have been using @jsdevtools/coverage-istanbul-loader through a Webpack configuration. However, I recently noticed that it messes up sourcemaps. The breakpoints are off and some valid lines you can't debug.

I want to try to babel-plugin-istanbul to see if it works better with sourcemaps.

[edumserrano]

To reiterate on @cenfun's comment, you can use the monocart-reporter to get code coverage in istanbul without having to instrument your code with babel-plugin-istanbul.

You can follow Playwright's documentation on code coverage to collect code coverage in V8 and and then use the monocart-reporter to convert the V8 code coverage to istanbul via the the toIstanbul code coverage option.

I've had issues with incorrect code coverage on my project using the v8-to-istanbul library shown on Playwright's documentation. See istanbuljs/v8-to-istanbul#216.

@cenfun fixed the V8 to istanbul conversion issue on monocart-reporter by removing the dependency on the v8-to-istanbul and writing something to replace it. See cenfun/monocart-reporter#50. Since the fix, I've had no issues with incorrect code coverage using it.

UPDATE:
See this comment for detailed guidance and code demo on setting up code coverage.

[michaelhays]

As an update, the situation here has gotten much less dire in the months since @edumserrano's June 4 comment, thanks to the excellent work by @cenfun on monocart-reporter and monocart-coverage-reports. I seriously cannot thank you enough for your valiant efforts there!

For those of you using Next.js, I made an example project, which should help serve as a starting point.

[edumserrano]

Indeed @cenfun's effort with monocart-reporter has been phenomenal.

With the recently release of v2 for the monocart-reporter it has become even easier to not only get a code coverage html report but also to get a code coverage report in different formats such as lcov and cobertura.

Like @michaelhays, I have recently created a set of sample projects on the edumserrano/playwright-adventures repo that should help people take advantage of the monocart-reporter v2 release to get code coverage. See:

• Playwright code coverage with monocart-reporter
• Playwright code coverage with Istanbul via Webpack Babel plugin

Hopefully these demos will provide enough guidance for people to get code coverage for their projects.

[stevez]

@cenfun, @michaelhays, monocarg-reporter looks very interesting, I am going to try it. I have one question: does it support server code coverage? We are using NextJS and have the server side code using express

[michaelhays]

^ Server-side coverage is being tracked at cenfun/monocart-coverage-reports#1, for those interested