Why is this exception not handled inside parse_result_file()? If the exception can be handled as is done here, it should be handled where it is first encountered; otherwise every caller of parse_result_file needs to implement it. Seems the results can be handled quite symmetrically to the "normal" return values, which would be even more reason to move the try … except inside.

Done with next push.

If we mark the "Failed tests" entry also in the "Errors" column, we could keep len(results["Failed tests"]) == results["Errors"] + results["Failures"], which seems like an easy to sustain symmetry and follows the principle of least surprise.

My preference would be to have the failed and errored tests separately.

When parsing of the XML file fails, we lose all information about an entire phase of the testing process. We do not even know how many tests were in this phase and could therefore be considered "Failed". Indeed, in all cases I have seen so far, the tests actually all passed, we just could not parse this because the XML file was corrupted.

Therefore, we need to handle parse failures differently and cannot include them in the other statistics.

The exact number of failures is not relevant, since the type of failures determines criticality and required actions. So I think it's ok to have parsing failure recorded as "one failure". The same happens in other cases where we treat one wrong result different from any number of other errors. If there is at least one failure in a stage we need to look into it and understand what this means anyway. It's not possible to dismiss a number of failures just because the number is small.

@lekshmideepu, yes, that's why there are Errors and Failures. My point is that I think we do not need a third type that requires extra condition checks when interpreting the results. For me it would be intuitively clean to assert correct function of the testsuite by an

assert results["Failures"] + results["Errors"] == len(results["Failed tests"])

Where the list of failed tests contains messages for each of the errors and failures. The level on which something went wrong is not known for each failure anyway. If a test fails, this can be a problem anywhere between external libraries, nest code or broken test or test-suite. This is not special for a parsing error.

@terhorstd Thanks for the clarification. I thought it differently. I agree with your point.

@terhorstd In Pytest, "error" and "failure" have specific meanings (see https://stackoverflow.com/questions/21513718/pytest-error-vs-fail): If an assertion is triggered while a test_...() is executed, this counts as a failure, while a failed assertion outside a test_, especially in fixtures, is counted as an error. The sum of passes, failures, errors and skips should always equal the number of tests in total. This should apply for each row in our table and the totals at the bottom.

If we cannot parse an XML results file, we have no information and just counting that as an error would break the logic of the PyTest result counts. Therefore, I still think that it is most sensible to treat XML parsing errors separately from PyTest result tabulation.

Not necessary if marked as in previous comment. The extra XML PARSING FAILURE string can be absorbed into msg and I don't see why centering necessary in addition to all-caps.

I have removed the all caps. As explained above, we need to treat this case special. And I find the centred message much more visually pleasing :).