How to extract a list of tests cases and their categories programmatically.

[UyttenhoveSimon]

Hello Folks,

This is a question.

I am trying to get a list of all the test cases and their respective categories.

Currently I see three options:

1. Reflection: going through all methods, check for TestAttribute/TestCaseAttribute/TestFixture...

2. Using a runner, extract the xml with runner.Explore and parse the xml.

3. Use what NUnit is already doing.

My question revolves around the latter:
How to "flatten" the categories for each testcase? (when categories are put on abstract class or the whole class)
Ideally I would like to reuse what is already in use in NUnit to extract its test list.
Any suggestion as to where to start reading ?

Best regards,
Simon

[shack05]

Hi,
I spent some time recently trying to accomplish the same and tried a few things before settling on my current solution (which may or may not work for you). I needed a fairly general purpose solution which would work for different nunit3 test projects (some targeting dotnet framework, and some targeting dotnet core). Initially I tried reflection and assembly loading, and invoked the BuildFrom() from the ITestBuilder interface. It worked for most, but not all projects, and it felt like a lot of work and I'd prefer to let NUnit do the heavy lifting.

Next I tried the NUnit engine api (scroll down a bit to see an example) and it worked for the nunit projects targeting dotnet framework, but I ran into issues loading dependencies for projects targeting dotnet core, but of course YMMV.

In the end I realised it would be nice if my solution also worked for test solutions using other unit testing frameworks so created a custom vstest test logger which I use along with the nunit3testadapter. The custom logger defines event handlers for the DiscoveredTests and DiscoveryComplete events. The logger serializes the list of discovered testcase objects to a json file which I then process elsewhere. To use the logger I run vstest console, providing the /ListTests switch so that only test discovery is run, and another option to use the custom logger: vstest.console.exe <testassemblypath> /ListTests /Logger:<customloggername> -- RunConfiguration.TestAdaptersPaths=<directoryofcustomlogger>;<directoryofnunitadapter> (note there is a space after the double hyphens). Just add a reference to Microsoft.TestPlatform.TranslationLayer because that gives you access to both the vstest object model and the JsonDataSerializer for serializing/deserializing the discovered test case data. I think you need at least v16 of the translationlayer package because that is compatible with the version of the platform which introduced the test discovery events. You can download vstest from nuget or you will already have it if you have visual studio 2019 installed.

Another alternative could be using vstest.console.exe and the NUnit3Adapter, and enabling the DumpXmlTestDiscovery adapter setting. However the dump is diagnostic info so its not really a public API and I suppose it could change. I haven't tried it but you should be able to use the same vstest.console.exe command line trick shown above to enable the dumpxmltestdiscovery option from the command line: vstest.console.exe <testassemblypath> /ListTests -- NUnit.DumpXmlTestDiscovery=True.

That solution may be a bit over the top for your needs, but it ended up not being much code at all. It just requires consumers to be using vstest version 16 or above. I can't remember off the top of my head, but I think the categories were 'flattened'.

References:

• Creating a custom logger
• Passing runsettings to vstest from the command line
• Vstest command line args

[edit] updated the example usage - I had included the incorrect runconfiguration property

[OsirisTerje]

The Xml from the DumpXmlTestDiscovery for the test output itself (!) comes directly from the NUnit Engine, and is not likely to change in itself. This is the same Xml information that the adapter uses, so any change is this would require a new adapter, and would break backwards compatibility, something we are very careful about. However, the file in itself will be expanded, so your code should take that into consideration. Grabbing the test discovery output nodes however, should be perfectly safe.

[UyttenhoveSimon]

Hello @shack05 and @OsirisTerje ,

Thanks a lot for your insights!

To recap all the solutions:

1. Reflection:
   Assembly x = Assembly.GetExecutingAssembly();
   var classes = x.GetExportedTypes();

      foreach (System.Type type in classes)
      {
          MethodInfo[] methods = type.GetMethods();
          foreach (MethodInfo methodInfo in methods)
          {
              if ((methodInfo.GetCustomAttributes(typeof(TestAttribute), true).Length == 1)
   

...

2. NUnit Engine API:

ITestEngine engine = TestEngineActivator.CreateInstance();

        // Create a simple test package - one assembly, no special settings
        TestPackage package = new NUnit.Engine.TestPackage(Assembly.GetExecutingAssembly().Location);

        // Get a runner for the test package
        ITestRunner runner = engine.GetRunner(package);

        var tree = runner.Explore(NUnit.Engine.TestFilter.Empty);//new NUnit.Engine.TestFilter("*"));

3. Consoles (either NUnit or vstest):
   nunit3-console.exe Tests.dll --explore:"nunit3-test.xml" --where="{testFilter}"

I will probably end up mixing 1 and 3.