Counter-intuitive (erroneous?) behavior with test groups

[fgrie]

Hi,
I have weird behaviour with the following test file:

TEST_GROUP(Validation_ShouldFail)
{
};

TEST_GROUP(Validation_ShouldPass)
{
};

TEST(Validation_ShouldFail, ShouldFail1)
{
    CHECK(false);
}

TEST(Validation_ShouldPass, ShouldPass1)
{
    CHECK(true);
}

TEST(Validation_ShouldPass, ShouldPass2)
{
    CHECK(true);
}

TEST(Validation_ShouldFail, ShouldFail2)
{
    CHECK(false);
}

Running these tests with -ojunit, I get the following output files:

• A cpputest_Validation_ShouldPass.xml, which looks as expected:

<?xml version="1.0" encoding="UTF-8" ?>
<testsuite errors="0" failures="0" hostname="localhost" name="Validation_ShouldPass" tests="2" time="0.002" timestamp="*censored*">
<properties>
</properties>
<testcase classname="Validation_ShouldPass" name="ShouldPass2" assertions="1" time="0.000" file="*censored*.cpp" line="22">
</testcase>
<testcase classname="Validation_ShouldPass" name="ShouldPass1" assertions="1" time="0.000" file="*censored*.cpp" line="17">
</testcase>
<system-out></system-out>
<system-err></system-err>
</testsuite>

• A cpputest_Validation_ShouldFail.xml, which only contains one test result:

<?xml version="1.0" encoding="UTF-8" ?>
<testsuite errors="0" failures="1" hostname="localhost" name="Validation_ShouldFail" tests="1" time="0.003" timestamp="*censored* ">
<properties>
</properties>
<testcase classname="Validation_ShouldFail" name="ShouldFail2" assertions="1" time="0.000" file="*censored*.cpp" line="27">
<failure message="*censored*.cpp:29: CHECK(false) failed" type="AssertionFailedError">
</failure>
</testcase>
<system-out></system-out>
<system-err></system-err>
</testsuite>

I tracked down the problem, and found that it lays in the test order:

TEST(Validation_ShouldFail, ShouldFail1);
TEST(Validation_ShouldPass, ShouldPass1);
TEST(Validation_ShouldPass, ShouldPass2);
TEST(Validation_ShouldFail, ShouldFail2);

Interleaving the tests leads to CppUTest not recognizing the groups correctly.
I assume that this behavior is not intended, as it also contradicts

cpputest/tests/CppUTest/TestRegistryTest.cpp

Line 331 in 81eb8b8

TEST(TestRegistry, listTestGroupNames_shouldListBackwardsGroup1AfterGroup11AndGroup2OnlyOnce)

It is caused by this line:

cpputest/src/CppUTest/TestRegistry.cpp

Line 67 in 81eb8b8

if (endOfGroup(test)) {

As the tests not executed in group order, endOfGroup(test) becomes true three times (instead of two).