Is there a way to enable test coverage with thresholds using the nx test executor?

[dasco144]

I've migrated an existing dotnet 5/angular app into a Nx mono repo, and most things seem to be working great.
One thing that I'm not sure about currently is how we can use the text executor with options for threshold validation, as well as coverage report generation (using the dotnet reportgenerator tool).

Previously, I had the following targets in my test project

<Target Name="TestWithCoverage">
  <Exec Command="dotnet test API.Tests.csproj  @coverlet-coverage.rsp" WorkingDirectory="./" />
</Target>

<Target Name="TestWithReport">
  <Exec Command="dotnet msbuild /t:TestWithCoverage" WorkingDirectory="./" ContinueOnError="WarnAndContinue" />
  <Exec Command="dotnet reportgenerator -reports:./TestResults/lcov.info -targetdir:./TestResults/report" WorkingDirectory="./" />
</Target>

with @coverlet-coverage.rsp as below

/p:CollectCoverage=true
/p:CoverletOutput=TestResults/lcov.info
/p:CoverletOutputFormat=lcov
/p:SkipAutoProps=true
/p:Threshold="95.61 %2c 94.9 %2c 94.67"
/p:ThresholdType="line %2c branch %2c method"

After moving into the Nx workspace, I've been trying to achieve similar using the nx test executor.

The closest I've got is the following:

angular.json with the api-test app configuration

"api-test": {
    "root": "apps/api-test",
    "projectType": "application",
    "sourceRoot": "apps/api-test",
    "architect": {
        "test": {
            "builder": "@nx-dotnet/core:test",
            "options": {
                "settings": "coverlet.runsettings.xml"
            }
        }
    }
}

with coverlet.runsettings.xml as below

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
    <RunConfiguration>
        <ResultsDirectory>..\..\coverage\api-test</ResultsDirectory>
        <TreatNoTestsAsError>true</TreatNoTestsAsError>
    </RunConfiguration>

    <DataCollectionRunSettings>
        <DataCollectors>
            <DataCollector friendlyName="XPlat code coverage">
                <Configuration>
                    <Format>lcov</Format>
                    <SkipAutoProps>true</SkipAutoProps>
                    <!-- <CollectCoverage>true</CollectCoverage>
                    <Threshold>95.61 %2c 94.9 %2c 94.67</Threshold>
                    <ThresholdType>line %2c branch %2c method</ThresholdType> -->
                </Configuration>
            </DataCollector>
        </DataCollectors>
    </DataCollectionRunSettings>
</RunSettings>

The problems that I am not sure how to solve are as follows:

1. Using a runsettings file, uses VSTest, which according to this doesn't support threshold validation and showing the results in the console.
2. How can I run the report generator command after the coverage file is generated? Previously I would run the TestWithReport target that would run both commands in order.

A possible solution would be if there is a way to run the coverlet with the switches defined in the @coverlet-coverage.rsp, but I'm not sure how I would configure this in the angular.json test target.

Not sure if I'm missing something obvious here.
Any ideas/help would be appreciated

Thanks

[AgentEnder]

There are a couple ways to solve this in the context of the nx monorepo. To get these to work as is, without changes on the nx-dotnet side, you can set some targets up in workspace.json / project.json files that use @nrwl/workspace:run-commands as the executor. The docs for that are here: https://nx.dev/l/a/workspace/run-commands-executor. You may use dotnet msbuild \t:TestWithReport as the command for example, with cwd set to the apps root.

I'll likely work on an msbuild executor in the near future, but would welcome a PR for one if anyone is watching 🚀

[dasco144]

Thanks @AgentEnder. The run commands worked for now with the below config.

"test": {
    "builder": "@nrwl/workspace:run-commands",
    "options": {
        "cwd": "apps/api-test",
        "commands": [
            {
                "command": "dotnet msbuild /t:TestWithReport",
                "forwardAllArgs": false
            }
        ]
    }
}

One thing is that the coverage would now run every time with this custom command set.
It would be great if we could toggle coverage from the nx run-many command.
Eg, if running
nx run-many --a --code-coverage
The code-coverage command would translate into running an equivalent command for dotnet test --codeCoverage.

I did try using args, but it doesn't look like that would have the ability to output specific text based on whether args.codeCoverage is set to true
eg.
"command": "dotnet {args.codeCoverage === true ? 'msbuild /t:TestWithReport' : 'test'}"
This could just be me not understanding how to change a command based on an args value. The examples do only seem to show simple interpolation of the value and not a statement that can run and return a result

[AgentEnder]

Yeah, as far as changing which target based on an argument thats not currently supported. I'd recommend the new target being called test-with-coverage or something similar and leaving the existing test target if there are times you would not want the coverage, or changing the default targets name to something like test-without-coverage if it is less used.

Either way, if you want caching to work properly be sure to setup the outputs for Nx and add the new target to cacheableOperations in nx.json

[dasco144]

Great, I will look into those options. Thank you for the help