Package information is lost in the Surefire XML reports when using JUnit 5 with Maven

[SlavikZ]

When using ArchUnit @AnalyzeClasses annotation with JUnit 5 run by Maven surefire plugin, the generated XML test results are missing the package name in the classname attribute of the testcase tag. When using JUnit 4 run by Maven or any JUnit run by Gradle - classname contains full class name.

The problem could be easily reproduced on a forked ArchUnit examples project with added maven support: https://github.com/SlavikZ/ArchUnit-Examples/tree/with-maven
If run JUnit5 examples with ./mvnw test -pl ./example-junit5 and check example-junit5/target/surefire-reports/TEST-com.tngtech.archunit.example test.junit5.CodingRulesTest.xml report file, the testcase tags will look like this:

<testcase name="no_access_to_standard_streams" classname="CodingRulesTest" time="0.058">

In the case when run the same tests with Gradle - the appropriate line in the example-junit5/build/test-results/test/TEST-com.tngtech.archunit.exampletest.junit5.CodingRulesTest.xml report file will look like this:

<testcase name="no_access_to_standard_streams" classname="com.tngtech.archunit.exampletest.junit5.CodingRulesTest" time="0.023">

[lrozenblyum]

We're also affected by this issue which makes the JUnit report partially corrupted - ArchUnit tests go to the (root) category instead of the proper package.

[ls-jad-elkik]

I noticed that by setting this configuration:

  <statelessTestsetReporter implementation="org.apache.maven.plugin.surefire.extensions.junit5.JUnit5Xml30StatelessReporter">
      <disable>false</disable>
      <version>3.0</version>
      <usePhrasedFileName>false</usePhrasedFileName>
      <!-- Using @DisplayName for test class names will cause issues with Circle CI timing detection -->
      <usePhrasedTestSuiteClassName>false</usePhrasedTestSuiteClassName>
      <usePhrasedTestCaseClassName>false</usePhrasedTestCaseClassName>
      <usePhrasedTestCaseMethodName>true</usePhrasedTestCaseMethodName>
  </statelessTestsetReporter>

(Notice usePhrasedTestSuiteClassName and usePhrasedTestCaseClassName being set to false)

You will start seeing fully qualified class names in the testsuite's name and the testcase's classname.

There was still an issue when using ArchTests.in(SomeOtherClass.class) where the testsuite and testcase would be reported under SomeOtherClass instead of the one where ArchTests.in is called.

[codecholeric]

If the report is correct using Gradle, but broken using the Maven Surefire plugin, doesn't that point to a bug in the Maven Surefire plugin's JUnit 5 support? 🤔 Why do you think this is something to fix in ArchUnit rather than Surefire?