Issues retrieving code coverage for a large codebase #5554
Comments
|
Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support. |
|
moved this since it's in the vscode team's apex library |
|
@amtrack we are aware of the issue around volume and additional time to process code coverage. I see that you have already tried bumping max heap space to no avail. I also see that the machine being used for CI/CD is not very robust. Given the limited memory on the CI/CD machine, out recommendation would be to breakup the test runs into smaller groups. The CLI supports running of Apex Test Suites, as well as taking a list of Apex test classes to run. Thank you for the example project that challenges both time and space dimension of this issue. We also use similar configurations for our testing. The largest of those produces almost a million code coverage records and requires a heap space of 32GB. We are looking into alternatives to help minimize the memory consumption. |
|
@peternhale Thanks for your detailed response! I'm wondering if you at Salesforce could eventually make the Apparently this is possible for the MetadataComponentDependency Tooling API object:
|
|
@amtrack It would be nice to have the ApexCodeCoverage object available in Bulkv2. I don't believe either bulk of the endpoints support tooling objects. As for improving fetch time, the Coverage field is blob of sorts, that stores two arrays, covered and uncovered lines. When included in a query, the fetch times go up dramatically, due to the field's complexity. |
I honestly believe the same but the docs say the Bulk API 2.0 does support the I wanted to test it myself but failed quickly failed: $ sf data query -q "SELECT Id FROM MetadataComponentDependency" --use-tooling-api --bulk
Error (2): The following errors occurred:
--bulk=true cannot also be provided when using --use-tooling-api
--use-tooling-api=true cannot also be provided when using --bulk
See more help with --helpLooking at the code in the |
|
The errors you are seeing are from the cli query command, rather than to bulk api. Until now, bulk and tooling don'e mix. |
|
@amtrack have you made any progress in getting the tests w/coverage to run on your CI/CD machine? |
|
@peternhale thanks for asking! After trying lots of things and reading https://developer.salesforce.com/blogs/developer-relations/2012/11/how-code-coverage-works we're probably going to switch to Aggregated Code Coverage. |
|
@amtrack Thanks for getting back to me. We do have work on backlog to try to better handle large test runs. Please file the issue you mentioned when you have the details gathered. I would like to close this issue if it is ok with you. |
|
@peternhale OK, I'm looking forward to test it again when you have better handling of large test runs implemented. |
Summary
Retrieving the code coverage for a large codebase is very slow and uses a lot of RAM.
In our case, the unit tests only take 20 minutes to run, but retrieving the code coverage takes 30 minutes and even fails with a
JavaScript heap out of memoryerror in some environments.Steps To Reproduce
Create a Scratch Org and deploy some generated Apex Classes (see this repository).
Run the unit tests:
sf apex run test -l RunLocalTests --output-dir test-results --wait 60 --code-coverageExpected result
After the unit tests have finished, I expect it will not take more than 5 minutes to retrieve the code coverage.
Actual result
It either takes very long (30 minutes) to retrieve the code coverage or even fails with a
JavaScript heap out of memoryin some environments.System Information
{ "architecture": "darwin-arm64", "cliVersion": "@salesforce/cli/2.37.4", "nodeVersion": "node-v20.10.0", "osVersion": "Darwin 23.4.0", "rootPath": "/Users/john.doe/.config/yarn/global/node_modules/@salesforce/cli", "shell": "zsh", "pluginVersions": [ "@oclif/plugin-autocomplete 3.0.13 (core)", "@oclif/plugin-commands 3.2.2 (core)", "@oclif/plugin-help 6.0.20 (core)", "@oclif/plugin-not-found 3.1.2 (core)", "@oclif/plugin-plugins 5.0.7 (core)", "@oclif/plugin-search 1.0.20 (core)", "@oclif/plugin-update 4.2.4 (core)", "@oclif/plugin-version 2.0.16 (core)", "@oclif/plugin-warn-if-update-available 3.0.15 (core)", "@oclif/plugin-which 3.1.7 (core)", "@salesforce/cli 2.37.4 (core)", "apex 3.1.4 (core)", "auth 3.5.5 (core)", "data 3.2.5 (core)", "deploy-retrieve 3.5.6 (core)", "info 3.1.3 (core)", "limits 3.2.1 (core)", "marketplace 1.1.1 (core)", "org 4.0.4 (core)", "packaging 2.3.0 (core)", "schema 3.2.1 (core)", "settings 2.1.2 (core)", "sobject 1.2.2 (core)", "source 3.2.3 (core)", "telemetry 3.1.18 (core)", "templates 56.1.1 (core)", "trust 3.5.4 (core)", "user 3.4.3 (core)" ] }Additional information
This is probably a consequence of fixing an issue with incomplete code coverage report which was released in
@salesforce/cliv2.22.7.We didn't notice the incomplete coverage reports but I can now reproduce this with an old version.
Debug logs from a Mac when retrieving the code coverage succeeds but takes 30 minutes:
JavaScript heap out of memoryerror on a CI machine with 8GB RAM and heap size increased from 2GB default to 4GB usingNODE_OPTIONS=--max_old_space_size=4096.The text was updated successfully, but these errors were encountered: