Issue in parsing cobertura-coverage.xml

[ankit-arora12]

For cobertura-coverage.xml files, coverage reported by ReportGenerator is different from what we see in the xml file.
For the attached coverage file, coverage reported by

• reportGenerator: 38.8% coverage (108 out of 278 branches)
• as per the coverageFile: 37.415% coverage (110 out of 294 branches)

Can you please look into why this is happening?

cobertura-coverage.txt

I've changed the extension of cobertura-coverage.xml to .txt as github didn't support attaching .xml files.

[danielpalme]

I don't know how the values in the Cobertura file are calculated.

ReportGenerator calculates them in the following way:
Each line with a condition-coverage attribute contains branches.

In the following sample: condition-coverage="75% (3/4)"
3 of 4 branches are covered.

I just extracted the lines with condition-coverage attributes and summed them up.
In total I get 278 branches and 108 of them are covered.
So ReportGenerator's calculation seems to be correct.

[danielpalme]

Here are the extracted values:

Covered	Total
1	2
2	4
3	4
3	4
0	4
0	2
0	2
0	2
0	5
0	2
0	2
0	2
0	2
0	2
0	2
0	2
2	2
2	2
4	4
2	2
4	4
2	2
1	2
0	2
2	2
2	2
2	2
0	2
0	2
2	2
2	2
1	2
1	2
1	2
1	2
2	2
2	2
2	2
2	2
2	2
7	7
2	2
4	4
2	2
2	2
1	2
1	2
2	2
1	2
2	2
2	2
1	2
2	2
1	2
2	2
2	2
1	2
2	2
2	2
1	2
3	4
2	2
1	2
0	2
4	4
2	5
2	2
2	2
0	2
1	2
1	2
0	4
0	4
0	2
0	2
0	2
0	2
0	4
0	2
0	2
0	2
0	2
0	2
0	4
0	4
0	4
0	2
0	2
0	2
0	2
0	2
0	2
0	2
0	2
0	2
0	2
0	2
0	2
0	2
0	2
0	2
0	2
0	2
0	2
0	4
0	2
0	2
0	2
0	2
0	2
0	2
0	5
0	2
0	2
0	2
0	2
108	278

[ankit-arora12]

In that case, can we say that there is an issue with coverage file itself because in the summary at the top branches covered and branches total are diff?

[ReneSchumacher]

Hi @danielpalme,

I am not exactly sure, which tool is responsible for the issue, but in my case, ReportGenerator reports wrong data. It simply ignores all switch statements in the code. However, the corresponding cobertura report has no indication of the switch statement. Thus, I guess Istanbul is generating an incomplete report.

Here's a quick example:

The code (TypeScript) looks like this:

import Result from '../../Result';
import PolicyBase from '../PolicyBase';
import IRule from '../rules/IRule';
import FixedCoverageThresholdRule from './rules/FixedCoverageThresholdRule';
import CompareToBaselineCoverageRule from './rules/CompareToBaselineCoverageRule';
import CoverageValueReader from './CoverageValueReader';
import CoverageValue from './CoverageValue';

export default class CodeCoveragePolicy extends PolicyBase {
    public async run(): Promise<Result> {
        this.result.logger.log(this.parameters.getLocalizedMessage('EvaluateCoverage'));

        let coverageReader = new CoverageValueReader(this.parameters, this.result);
        let coverageValue = await coverageReader.getCoverageValue(this.parameters.getTaskVariable('System.TeamProject'), Number(this.parameters.getTaskVariable('Build.BuildId')), true);

        // The CoverageValueReader might fail the policy if explicit filtering is enabled and no data is found
        if (this.result.failed) {
            return this.result;
        }
        
        let policyRule: IRule<CoverageValue>;
        switch (this.parameters.getTaskInput('coverageFailOption', true).toUpperCase()) {
            case 'FIXED':
                policyRule = new FixedCoverageThresholdRule(this.parameters, this.result, coverageValue);
                break;
            case 'BUILD':
                policyRule = new CompareToBaselineCoverageRule(this.parameters, this.result, coverageValue);
                break;
            default:
                this.result.error(this.parameters.getLocalizedMessage('InvalidParameterValue', this.parameters.getTaskInput('warningFailOption', true), 'Fail Build On (coverageFailOption)'));
                break;
        }
        if (policyRule && await policyRule.evaluate()) {
            this.result.success(this.parameters.getLocalizedMessage('CoveragePassed', coverageValue.percentage, coverageValue.elements, coverageValue.valueLabel));
        }

        return this.result;
    }
}

The report shows this:

<class name="CodeCoveragePolicy.ts" filename="src\policies\codeCoverage\CodeCoveragePolicy.ts" line-rate="0.9" branch-rate="0.8889">
  <methods>
    <method name="(anonymous_6)" hits="15" signature="()V">
      <lines>
        <line number="10" hits="15"/>
      </lines>
    </method>
  </methods>
  <lines>
    <line number="2" hits="1" branch="false"/>
    <line number="4" hits="1" branch="false"/>
    <line number="5" hits="1" branch="false"/>
    <line number="6" hits="1" branch="false"/>
    <line number="9" hits="1" branch="false"/>
    <line number="11" hits="15" branch="false"/>
    <line number="13" hits="15" branch="false"/>
    <line number="14" hits="15" branch="false"/>
    <line number="17" hits="15" branch="true" condition-coverage="100% (2/2)"/>
    <line number="18" hits="1" branch="false"/>
    <line number="22" hits="14" branch="false"/>
    <line number="24" hits="4" branch="false"/>
    <line number="25" hits="4" branch="false"/>
    <line number="27" hits="10" branch="false"/>
    <line number="28" hits="10" branch="false"/>
    <line number="30" hits="0" branch="false"/>
    <line number="31" hits="0" branch="false"/>
    <line number="33" hits="14" branch="true" condition-coverage="100% (4/4)"/>
    <line number="34" hits="7" branch="false"/>
    <line number="37" hits="14" branch="false"/>
  </lines>
</class>

ReportGenerator assumes 6 out of 6 branches (=100%) covered, but according to the report only 88.89% have been covered. Those 88.89% are 7 out of 9 branches and that is correct as in addition to the two if statements that account for the 6 branches there are three switch cases of which the default case is not covered.

I assume that line number 22 should have a conditions child element describing the switch cases. Unfortunately, I couldn't find a proper Cobertura report for a switch statement :(

@ankit-arora12: fyi

René

[danielpalme]

@ReneSchumacher
The Cobertura file only contains branch information for the if statements. So there is nothing that ReportGenerator can do about it.
Perhaps Instanbul is not able to handle switch statements correctly or it's just missing in the report.

See also: istanbuljs-archived-repos/istanbul-lib-instrument#31

[ReneSchumacher]

Hey,

Istanbul knows about the branches, hence the summary values are correct (see my sample above). Still, the Cobertura reporter in Istanbul does not report the line details correctly. I'm trying to get this sorted out. Imho, Istanbul is widely used for coverage generation for TypeScript and ReportGenerator is the de-facto standard for coverage report generation. I'd love to see them working together properly.

Here's the Istanbul issue: istanbuljs/istanbuljs#517

[ReneSchumacher]

Just for reference: I fixed the issue in istanbuljs and now am waiting for the maintainers to accept my PR istanbuljs/istanbuljs#518. With this, we'll see consistent outputs from Cobertura and ReportGenerator again :D