code coverage reporting is flakey

[JohnAlbin]

Package containing the bug

next-drupal (NPM package)

Describe the bug

If multiple jest files are run, the code coverage of previously-run tests can be reset during the Jest test run and erroneously report a smaller code coverage value than is truly covered. This faulty report is not consistently reported; sometimes the true coverage value is reported.

Expected behavior

The code coverage report is always consistent and accurate instead of frequently reporting missing coverage for code that is obviously covered by existing tests.

Steps to reproduce:

1. Run yarn test multiple times making slight code changes each time that shouldn't change the coverage value.
2. Notice the coverage value vacillate.
3. 😢

Additional context

See jestjs/jest#14766 and nodejs/node#51251

[JohnAlbin]

In nodejs/node#51251, someone mentions that the bug is related to the version of Node being used.

20.9.0 just works
20.10.0 fails

It might also be a bug in the collect-v8-coverage packages used by Jest. SimenB/collect-v8-coverage#235

[JohnAlbin]

GitHub CI is currently using node v18.20.1 where the code coverage is flakey and occasionally failing.

And it looks like v18 versions that were released before v20.10.0 was released (on 2023-11-22) might work. v18.18.x was released before that.

[kmccammon]

I posted this originally on: nodejs/node#51251

Reposting here in case it's helpful...

I have been facing the same issue: my coverage report was 100% with 20.9.0, but as of 20.10.0, it now reports less than 100% coverage and somewhat randomly flags lines not covered.

I noticed that I get the same coverage reporting flakiness with 18.20.2 (I initially saw this issue when my GitHub actions started to fail). Coverage works fine with 18.19.1.

So there seems to be a change between 18.19.1 and 18.20.0 that causes the issue as well as between 20.9.0 to 20.10.0.

This is now causing my GitHub CI/CD actions to fail since GitHub is using 18.20.2.

[JohnAlbin]

I tested v18.19 and it does not appear to be flakey.

[JohnAlbin]

@kmccammon Yep! That's exactly what I noticed as well. I've changed this project's .nvmrc to use v18.19 for now. Thanks for confirming!

I'm leaving this issue open until we can revert the work-around.