Feature: Add test-suite support to IReporter interface and JUnit-Reporter

[jenisys]

Description

Currently, the doctest::IReporter interface lacks methods to add test-suite information.

CURRENT STATE:

• The junit reporter just provides only <testsuite/> XML element(s) for the test-program that is executed
• JUnitXML supports nested testsuite(s), meaning a <testsuite/> XML element(s) within another <testsuite/> XML element.

BUT:

• Testsuites (as a general concept) provide a hierarchical grouping mechanism
• JUnitXML nested testsuites map well to the concept how testsuites are used in the C++ source code.

NOTES:

• IDE(s), like Jetbrains, etc. normally provide a TestRunner/TestExplorer where this hierarchical testsuite grouping is shown.

Steps to reproduce

MOTIVATING EXAMPLE:

// -- FILE: test_example_with_testsuites.cpp
#include "doctest/doctest.h"

TEST_SUITE("test_suite_1") {
    TEST_CASE("one")
    {
        MESSAGE("Part of testsuite_1: test=one");
    }

    TEST_CASE("two")
    {
        MESSAGE("Part of testsuite_1: test=two");
    }
}

TEST_SUITE("test_suite_2") {
    TEST_CASE("one")
    {
        MESSAGE("Part of testsuite_2: test=one");
    }

    TEST_CASE("two")
    {
        MESSAGE("Part of testsuite_2: test=two");
    }
}

When you build the test program and run it with ... --reporters=junit, you get the following JUnitXML report:

<!-- FILE: junit_test_report.xml -->
<testsuites>
  <testsuite name="test_example" errors="0" failures="0" tests="4" ... doctest_version="2.4.11">
    <testcase classname=".../test_testsuite.cpp" name="one" time="0" status="run"/>
    <testcase classname=".../test_testsuite.cpp" name="two" time="0" status="run"/>
    <testcase classname=".../test_testsuite.cpp" name="one" time="0" status="run"/>
    <testcase classname=".../test_testsuite.cpp" name="two" time="0" status="run"/> 
  <testsuite>
<testsuites>

PROBLEM:

• Currently, it is completely unclear why tests "one" and "two" occur twice
• Testsuite context is lost in the report

DESIRED OUTPUT:

<!-- FILE: junit_test_report.xml -->
<testsuites>
  <testsuite name="test_example" errors="0" failures="0" tests="4" ... doctest_version="2.4.11">
    <testsuite name="test_suite_1" errors="0" failures="0" tests="2" ... >
      <testcase classname=".../test_testsuite.cpp" name="one" time="0" status="run"/>
      <testcase classname=".../test_testsuite.cpp" name="two" time="0" status="run"/>
    </testsuite>
    <testsuite name="test_suite_2" errors="0" failures="0" tests="2" ... >
      <testcase classname=".../test_testsuite.cpp" name="one" time="0" status="run"/>
      <testcase classname=".../test_testsuite.cpp" name="two" time="0" status="run"/> 
    </testsuite>
  </testsuite>
</testsuites>

EXAMPLE 2: Test suite name provides context info for test-cases

// -- FILE: test_example2_with_testsuites.cpp
#include "doctest/doctest.h"

TEST_SUITE("Function A") {
    TEST_CASE("It should succeed if ...") { ... }
    TEST_CASE("It should fail if ...") { ... }
    ...
}

TEST_SUITE("Function B") {
    TEST_CASE("It should succeed if ...") { ... }
    TEST_CASE("It should fail if ...") { ... }
    ...
}

CONSIDERED ALTERNATIVES: TEST_CASE/SUB_CASE combination

• This disambiguates the example above
• This combines the TEST_CASE name with the SUB_CASE name in one <testcase name="{TEST_CASE.name}/{SUBCASE.name}" /> XML element
• The testcase.name can become rather large
• BUT: The grouping gets slightly lost or must be rediscovered from the name info.

Extra information

DESIRED CHANGE:

• Add methods to doctest::Reporter to add testsuite entry/exit.
• Default implementation of these methods could be empty (to avoid to break existing reporters).
• Call theses methods when a testsuite entry/exit is encountered by the doctest framework

NICE TO HAVE:

• An extended junit reporter implementation that uses this additional testsuite context information to create the test report with the desired information

VERSION INFO:

• doctest version: v2.4.11