Code Coverage Reports in Deploy Commands Produce Unreliable Covered Lines

[mcarvin8]

Summary

This has already been reported as bugs such as forcedotcom/cli#1568 but this has been closed due to "working as expected".

Please look into this soon for a new CLI Build.

When running sf project deploy start/sf project deploy validate commands with the code coverage flags (like --coverage-formatters json), the uncovered lines reported back are correct. The covered lines are incorrect for various reasons:

• covered lines will sometimes be out-of-range lines (if there is a 98 line Apex Class, the coverage reports will report line 100 as covered)
• covered lines are inconsistent and report back more lines than there are in the file

Please update the Salesforce CLI to reliably report covered lines in the code coverage files so they can be exported to tools like SonarQube for code coverage tracking. In the current state, the uncovered lines can be exported to SonarQube correctly, but you need to have covered lines reported to SonarQube to calculate the coverage percentage. At this point, coverage in SonarQube will always be at 0% or the coverage % will be unreliable due to the covered lines being incorrectly reported.

I'm unclear if the CLI's apex commands produce the same outputs as the deploy commands in terms of code coverage. But we need reliable deploy coverage output so we can report back accurate code coverage during deployment and validation.

@xL3o - Please add anything I missed here based on our testing in my plugin.

See additional notes on my plugin here - mcarvin8/apex-code-coverage-transformer#2

Steps To Reproduce

1. Deploy a package with apex classes or triggers with the code coverage formatted as a JSON or JSON summary. sf project deploy start -x manifest/package.xml --coverage-formatters json
2. Confirm the covered and uncovered lines in the coverage JSON file created by the CLI

Expected result

Uncovered and covered lines are reported back correctly. The covered lines should not have out-of-range line numbers.

Actual result

Uncovered lines are correct. Covered Lines are incorrectly reported back.

System Information

Tested on Windows 11/Powershell and Ubuntu/Bash (CI/CD).

{
  "architecture": "win32-x64",
  "cliVersion": "@salesforce/cli/2.30.8",
  "nodeVersion": "node-v20.11.1",
  "osVersion": "Windows_NT 10.0.22621",
  "rootPath": "C:\\Users\\matthew.carvin\\AppData\\Local\\sf\\client\\2.30.8-d9ea14b",
  "shell": "cmd.exe",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 3.0.9 (core)",
    "@oclif/plugin-commands 3.1.4 (core)",
    "@oclif/plugin-help 6.0.13 (core)",
    "@oclif/plugin-not-found 3.0.11 (core)",
    "@oclif/plugin-plugins 4.2.5 (core)",
    "@oclif/plugin-search 1.0.16 (core)",
    "@oclif/plugin-update 4.1.13 (core)",
    "@oclif/plugin-version 2.0.12 (core)",
    "@oclif/plugin-warn-if-update-available 3.0.12 (core)",
    "@oclif/plugin-which 3.1.0 (core)",
    "@salesforce/cli 2.30.8 (core)",
    "apex 3.0.25 (core)",
    "auth 3.3.13 (core)",
    "data 3.1.1 (core)",
    "deploy-retrieve 3.2.16 (core)",
    "dev 2.1.13 (user)",
    "info 3.0.27 (core)",
    "limits 3.1.10 (core)",
    "marketplace 1.0.25 (core)",
    "org 3.3.14 (core)",
    "packaging 2.1.10 (core)",
    "schema 3.1.3 (core)",
    "settings 2.0.27 (core)",
    "sobject 1.1.13 (core)",
    "source 3.1.15 (core)",
    "telemetry 3.1.13 (core)",
    "templates 56.0.16 (core)",
    "trust 3.3.10 (core)",
    "user 3.2.11 (core)",
    "sfdx-decomposer 3.1.0 (user)",
    "sfdx-git-delta 5.34.0 (user)"
  ]
}

Additional information

[github-actions]

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

[shetzel]

There is nothing new to report on the code coverage front from that last issue you mentioned. The CLI team is dependent on some other internal teams and their priorities for any work to begin on this. If the code coverage data in the API response is not accurate then there's nothing we can do except report what's given to us. So from a CLI perspective, the code is making the correct calls and reporting the response as provided (i.e., working as designed), and to change that we need other teams to prioritize that work. It's on their roadmaps but I don't know when it will be delivered.

[mcarvin8]

Should we keep this issue open @shetzel or just close it since this seems out of the CLI team's responsibility? Seems like once the API team updates the API response to provide accurate code coverage, it will work fine on any version of the CLI.

[shetzel]

In my opinion we should close it. Unless there is a breaking change involved with responses there is nothing the CLI would have to do to provide better, more accurate code coverage when those fixes are made to the server side APIs and libraries. You and any others are welcome to post here asking for updates on this. I'm hopeful that the work will at least be prioritized this year and we can relay that information when we have it to keep everyone up to date.

...it will work fine on any version of the CLI.

If the fixes are only serverside, then yes. If there is more involved (e.g., updates to the apex-node library) then the CLI could need to bump version dependencies.

[jfaderanga]

it is a blocker for pipeline with Quality Gates. I've been struggling with this issue for few months now. One thing I noticed is the code should be modularized and clean for the issue not to happened. There are times that the error still shows up even the code is modularized and well written which becomes a blocker on development as the code can't be enhanced anymore.

Please re-open the issue. @mcarvin8 @shetzel

[mcarvin8]

@jfaderanga - Per @shetzel , this issue needs to be fixed by the team that manages the API first. I agree with you that this is a blocker for anyone who is trying to implement Quality Gates that enforce code coverage.

@shetzel - Is there a different way to contact the right team that owns updates to the API (outside of a general Salesforce support case)?

[shetzel]

I agree with both of you that it's an issue, just not an issue that anyone on the CLI team can do anything about. This repo is about issues with the CLI. If it's open, it implies we can take some action on it but in this case we can't.

Is there a different way to contact the right team that owns updates to the API (outside of a general Salesforce support case)?

The best ways to influence the right team are opening a support case, and voting up the idea on the IdeaExchange.

[mcarvin8]

it is a blocker for pipeline with Quality Gates. I've been struggling with this issue for few months now. One thing I noticed is the code should be modularized and clean for the issue not to happened. There are times that the error still shows up even the code is modularized and well written which becomes a blocker on development as the code can't be enhanced anymore.

Please re-open the issue. @mcarvin8 @shetzel

mcarvin8/apex-code-coverage-transformer#3

@jfaderanga - I was working on this with my plug-in that converts the coverage JSON to an XML SonarQube will accept.

This will convert covered lines and work around this known issue by reassigning out of bound lines with unused lines. The coverage % will vary but it should still pass if you're hitting coverage. I could release this but I'm skeptical since the covered lines won't always report back the same way (comments and new lines can change it). But then again, however the API calculates deploy coverage is incorrect so this plugin would just convert that JSON entry to a XML entry.

Like I've seen it report back 86% coverage and then 88% coverage on other runs for example. So it should pass in most cases as long as your not near the 75% minimum requirement but this updates isn't ideal until the API is fixed to report back the right covered lines.

If you want, you can try my beta plugin with these updates - sf plugins install apex-code-coverage-transformer@beta

That creates a coverage.xml you can pass to SQ.

It's not ideal until the API is fixed but this will get coverage % in SonarQube relative to what the deploy command currently splits out.

[jfaderanga]

@mcarvin8 I'm using the direct XML (cobertura) output from the deploy command, it has coverage rate from the class node. I upload this to Azure DevOps and Sonar Cloud, from what I noticed, the SC only reads the coverage rate from the class node and it just uses the lines for the UI line highlighting. I haven't seen the JSON output but I expect it should be the same, does your plugin also update the coverage rate, it shouldn't be ay, just the lines?

I've been doing heavy manipulation already to fix the XML to correct the file path (no-map/classname issue), adding more may affect the pipeline performance, but I'll give it a go as I don't have any workaround at the moment, thanks for the suggestion.

[mcarvin8]

@jfaderanga - My plugin essentially just converts the JSON output from the deploy command into a XML file containing the uncovered lines/covered lines.

The latest build of my plugin just prints uncovered lines in the XML since that's accurate.
The beta build of my plugin prints covered and uncovered lines with some extra tinkering done on covered lines due to this bug.

All builds will take care of replacing no-map in the file name based on the --dx-directory flag (the default path being force-app/main/default). So my plugin would save you some steps that you're doing to fix the file-paths.

We noticed with SonarQube that just providing uncovered lines will not update the Coverage % (with the latest plugin build).

After doing the updates in the beta build with adding covered lines (automatically adjusting line numbers with this bug if the line number provided is out-of-range), we did confirm SonarQube calculates the coverage % automatically. It's just not as ideal as having the API return the correct # of covered lines back to the CLI so that my plugin can just convert the JSON output exactly as provided.

[xL3o]

The best ways to influence the right team are opening a support case, and voting up the idea on the IdeaExchange.

@shetzel is IdeaExchange really the best way forward in our situation? I mean this code coverage functionality for validation deployments is already part of the end product, rather than being a new "Idea" that the POs would need to prioritize. Therefore, there should be a more streamlined approach how to highlight this issue/bug directly to the correct team for them to start bug fixing.

[shetzel]

@xL3o - I meant that more as a general statement of how to get issues in front of non-CLI teams. For this issue, a support case is best. However, I'll open this back up and transfer it to the apex-node library. That team is a better choice to own this issue.

[jfaderanga]

@jfaderanga - My plugin essentially just converts the JSON output from the deploy command into a XML file containing the uncovered lines/covered lines.

The latest build of my plugin just prints uncovered lines in the XML since that's accurate. The beta build of my plugin prints covered and uncovered lines with some extra tinkering done on covered lines due to this bug.

All builds will take care of replacing no-map in the file name based on the --dx-directory flag (the default path being force-app/main/default). So my plugin would save you some steps that you're doing to fix the file-paths.

We noticed with SonarQube that just providing uncovered lines will not update the Coverage % (with the latest plugin build).

After doing the updates in the beta build with adding covered lines (automatically adjusting line numbers with this bug if the line number provided is out-of-range), we did confirm SonarQube calculates the coverage % automatically. It's just not as ideal as having the API return the correct # of covered lines back to the CLI so that my plugin can just convert the JSON output exactly as provided.

The no-map replacement feature, does it support multiple packageDirectories or just defaults it to force-app?

[mcarvin8]

Currently it just supports one directory via the --dx-directory flag. That flag defaults to force-app/main/default if you don't provide the flag a different path.

I can look into supporting multiple package directories if you want (probably something like scanning for the sfdx-project.json file and reading all directories listed).

[mcarvin8]

@jfaderanga - I just released a new version of the plugin which supports multiple package directories as listed in your project's sfdx-project.json file. If you run this plugin in the root directory of your project which has the JSON file, you don't need to supply the flag for the sfdx-project.json file path.

https://github.com/mcarvin8/apex-code-coverage-transformer/releases/tag/v1.5.0

This version (which is the latest/stable tag) only will add uncovered lines in the final XML file since that's what's always accurate.

I will need to do some additional work to back-fit these changes into a pre-release/beta build that adds covered lines.

Of course, if this issue is escalated and fixed soon, I don't need to do extra manipulation to correct covered lines as printed by the deploy command.

[jfaderanga]

@mcarvin8 that's awesome mate, thanks for the quick turn around time for this, i'll try this today..

[mcarvin8]

@jfaderanga - Awesome!

I just released a new beta pre-release build which will add covered lines to the final XML file.

Big catch like I've stated previously is that the Salesforce CLI will produce out-of-range covered lines (line 100 in a 98 line file might show as covered). To correct this, I added code that counts the total number of lines in the file and re-assigns out-of-range lines to an unused line number. It's not ideal (since @xL3o has demonstrated that the Salesforce CLI will produce extra lines in the coverage report as well - for example: Salesforce CLI will report 118 lines total covered & uncovered in a file that is only 100 lines). But this will ensure SonarQube always calculates Code Coverage percentage.

If you want to try the beta build with covered lines, run sf plugins install apex-code-coverage-transformer@1.6.0-beta.1 and give that a shot.

Otherwise, the latest build (1.5.0) is the safe route since that just prints uncovered lines which is always correct. SonarQube just won't calculate code coverage % if you only provide uncovered lines, which @xL3o and I have confirmed with previous builds.

So just to sum up:

1. Install sf plugins install apex-code-coverage-transformer@1.5.0 to produce a coverage XML with uncovered lines for multiple package directories.
   1. You will see uncovered lines in SonarQube when viewing the file, but you will not get code coverage percentage
2. Install sf plugins install apex-code-coverage-transformer@1.6.0-beta.1 to produce a coverage XML with covered and uncovered lines for multiple package directories, keeping in mind that Salesforce produces unreliable covered lines.
   1. You will get code coverage % in SonarQube, it just won't be 100% accurate due to how the API produces covered lines

[jfaderanga]

@mcarvin8 I have tried the beta version 1.6.0-beta.1 and it works as described.

From what I noticed with the API output, the coverage percentage is correct. If you divide the number of uncovered lines by the number of covered lines, it gives you the coverage percentage which is the same with what is showing in Salesforce.

The plugin beta version 1.6.0-beta.1 converts the JSON (api output) to XML with the exact number of covered and uncovered lines as what's in the JSON (assigning out-of-range lines to unused lines starting from line 1). This is great as this still gives the same coverage percentage assuming the number out of range lines won't go beyond the number of unused lines, which I think impossible to happen.

Also the plugin correctly maps the file path, kudos for this mate!

I would say your plugin mate is an acceptable long-term workaround for this issue as I bet it will take a long time for the issue to be generally fixed. I'm happy to implement this in our CI, thanks a lot @mcarvin8 and @xL3o and @shetzel ! Let me know when you release the stable version of the 1.6.0 beta.

[mcarvin8]

@jfaderanga - Thank you for testing this out and letting me know!

I have released the 1.6.0 tag which adds support for covered lines by re-numbering out-of-range lines.

Once this issue is resolved by Salesforce, I'll remove that re-number function.

sf plugins install apex-code-coverage-transformer@1.6.0

[jfaderanga]

@mcarvin8 I started looking into implementing this and made few more tests. The plugin seems to be having an issue trying to find a matching file if the file name contains underscores.

The sample below is from the JSON API output.

    "no-map/ABC_API_CLASSNAME": {
        "fnMap": {},
        "branchMap": {},
        "path": "no-map/ABC_API_CLASSNAME",
        "f": {},
        "b": {},
        "s": {
        ....

The plugin returns the following error even though the file is indeed within one of the package directory.
Error (1): The file name ABC_API_CLASSNAME was not found in any package directory.

[mcarvin8]

@jfaderanga Can you make an issue here on my repo, please? - https://github.com/mcarvin8/apex-code-coverage-transformer/issues

I just tested it out by renaming the Test Apex Class I use in my Unit Test to ABC_API_CLASSNAME.cls and I wasn't able to replicate this error in Windows. I also wasn't able to replicate this failure in my pipeline, which is Ubuntu. mcarvin8/apex-code-coverage-transformer@01012a7

What's the full path to the file in your package directory? In the code, I'm assuming the classes, triggers, and flows are in these sub-folder names (classes, flows, triggers). This works even if you use the default structure force-app/main/default or just 1 folder packaged. If this isn't the right assumption to make, I'll need to update the code differently.

I've only worked with repos following the default force-app/main/default structure so I'm unclear on how you can structure different package directories.

  let relativeClassPath = await findSubFolder(dxDirectory, 'classes');
  let relativeTriggerPath = await findSubFolder(dxDirectory, 'triggers');
  let relativeFlowPath = await findSubFolder(dxDirectory, 'flows');

EDIT: I have released a new version - sf plugins install apex-code-coverage-transformer@1.6.1 - which will recursively search each package directory to find a file match, without hard-coding any sub-folder names like classes, flows, or triggers.
I tested this in my team's repo (which is a pretty huge code base) and this still ran pretty quickly with the below JSON:

{
 "no-map/ACCOUNT_PROFILE": {
   "fnMap": {},
   "branchMap": {},
   "path": "no-map/ACCOUNT_PROFILE",

Please try 1.6.1 with the class you mentioned above and see if that resolves your issue.

[jfaderanga]

@mcarvin8 yeah looks like the 1.6.1 fixes it. I forgot to mentioned, the other directory in my project, the directory name has space in it. It's a messed up structure as it came from a unlock package.

Folder Name
  ﹂main
    ﹂ default
      ﹂classes
  ﹂Folder Name
     ﹂main
       ﹂ default
         ﹂ labels

The 1.6.1 seems fixed it, thanks heaps mate!

[mcarvin8]

@peternhale - Could you provide any updates to the task of fixing the API to correctly return Code Coverage covered lines during deploy commands? The covered lines returned by the API for deployment commands are inaccurate in their current state, which makes it unreliable to track coverage percentages in tools like SonarQube.

I recall your name being brought up in a code coverage discussion with Allan Oricil, so feel free to include anyone else here if you are the not right person to contact for this.

Please ignore the back-and-forth with JaysonF and I in previous comments on this thread regarding my Salesforce plugin which is designed to convert the code coverage JSON output to an XML for SQ.

[peternhale]

Hi @mcarvin8 Sure, the root issue of the inaccuracy of the code coverage for tests run during a deploy is in the data returned from the deploy. To get to the code coverage we drill down on the returned data starting with DeployResult -> DeployDetails[] -> RunTestResult -> CodeCoverageResult[]

You will find field locationsNotCovered on CodeCoverageResult, but no field for locationsCovered, which is where we find ourselves. The implementation tried to "fill in the blanks", but, as you know, does not do a good job inferring which are lines of code and which are not.

The CodeCoverageResult object needs to be changed to include covered locations. The object is owned by another team here at Salesforce. We have requested requested change, but the work has yet to be prioritized.

I have tried to find an alternative source to fetch the coverage for the test run, but that did not yield anything to use.