Exception when using test filters from .runsettings or --filter argument from dotnet test

[thomasbiel]

NUnit 4.0.1
NUnit3TestAdapter 4.5.0
dotnet 6.0 / windows 11
running tests from command line via "dotnet test"

when using a test filter like
(cat != UsesNetwork) and (cat != LongRunning)
in .runsettings file or via --filter argument for dotnet this exception occurs:

Exception NUnit.Engine.NUnitEngineException, Exception thrown executing tests in [...].dll
An exception occurred in the driver while counting test cases.
at NUnit.Engine.Runners.DirectTestRunner.CountTestCases(TestFilter filter)
at NUnit.Engine.Runners.MasterTestRunner.RunTests(ITestEventListener listener, TestFilter filter)
at NUnit.Engine.Runners.MasterTestRunner.Run(ITestEventListener listener, TestFilter filter)
at NUnit.VisualStudio.TestAdapter.NUnitEngine.NUnitEngineAdapter.Run(ITestEventListener listener, TestFilter filter) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnitEngine\NUnitEngineAdapter.cs:line 108
at NUnit.VisualStudio.TestAdapter.Execution.Run(TestFilter filter, DiscoveryConverter discovery, NUnit3TestExecutor nUnit3TestExecutor) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\Execution.cs:line 51
at NUnit.VisualStudio.TestAdapter.VsTestExecution.Run(TestFilter filter, DiscoveryConverter discovery, NUnit3TestExecutor nUnit3TestExecutor) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\Execution.cs:line 154
at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunAssembly(String assemblyPath, IGrouping2 testCases, TestFilter filter) in D:\repos\NUnit\nunit3-vs-adapter\src\NUnitTestAdapter\NUnit3TestExecutor.cs:line 295 InnerException: System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ArgumentOutOfRangeException: Index was out or range of valida values (Parameter 'index') Actual value was 0. at NUnit.Framework.Interfaces.TNode.NodeList.ThrowArgumentOutOfRangeException(Int32 index) at NUnit.Framework.Interfaces.TNode.get_FirstChild() at NUnit.Framework.Internal.TestFilter.FromXml(TNode node) at NUnit.Framework.Internal.TestFilter.GetChildNodeFilters(TNode node) at NUnit.Framework.Internal.TestFilter.FromXml(TNode node) at NUnit.Framework.Internal.TestFilter.FromXml(String xmlText) at NUnit.Framework.Api.FrameworkController.CountTests(String filter) --- End of inner exception stack trace --- at System.RuntimeMethodHandle.InvokeMethod(Object target, Span1& arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
at NUnit.Engine.Drivers.NUnitNetStandardDriver.ExecuteMethod(MethodInfo method, Object[] args)
at NUnit.Engine.Drivers.NUnitNetStandardDriver.ExecuteMethod(String methodName, Object[] args)
at NUnit.Engine.Drivers.NUnitNetStandardDriver.CountTestCases(String filter)
at NUnit.Engine.Runners.DirectTestRunner.CountTestCases(TestFilter filter)

[OsirisTerje]

The filter terms dont look right. Can you state the complete dotnet test command you're using?

A normal filter should look like:

dotnet test --filter TestCategory!=CategoryA&TestCategory!=CategoryB

Based on your filter command above, it may be that you're using a NUnit.Console Where statement, which also is supported by dotnet test, but then NOT using the --filter command, but the NUnit Where property instead. See Where syntax and the blog post here

For short, using the dotnet test, it would then look like:

dotnet test -- NUnit.Where="cat != CategoryA and cat != CategoryB"

PS: Note the space after --

PS2: If you add in the --filter it will override the Where, so dont do that.

---

If still not working, can you please upload a repro project?

You can a) Add it as a PR to https://github.com/nunit/nunit3-vs-adapter.issues, add it as a separate folder named IssueXXX where the XXX is the issue number, or b) Add it as a zipped attachment here , or c) Add it to your own repo and just post the link.

Make it as small as you can, it should compile "as is".

[thomasbiel]

I got the filter mixed up, I started with --filter TestCategory!=CategoryA&TestCategory!=CategoryB but that didn't work, so I read the documentation and tried the NUnit-specific version -- NUnit.Where="cat != CategoryA and cat != CategoryB", also w/o luck - regardless of adding the filter via CLI or in .runsettings. I will try to boil down a sample for you.

[thomasbiel]

Gotcha - everything works fine until there is a test marked with [Explicit] along with non-marked tests in a fixture.
NUnitFilterSample.zip
just run the test.ps1 und NUnitFilterSample\Product

[OsirisTerje]

What happens when there is an Explicit there?

[thomasbiel]

Then you get the mentioned exception, otherwise the testrun completes

[OsirisTerje]

Thanks @thomasbiel . I can repro this. I made the sample you added a bit simpler, and it seems that it crashes with only the single test class with 2 tests, one explicit, and one not.

Also, it crashes when the filter has two categories, but not with only one.
And the Explicit triggers it, remove that and it works.

I added dump files, and the filter line that is sent to the framework is:

<filter><and><not><cat>Sample</cat></not><not><cat>Stress</cat></not></and><not><prop name='Explicit'>true</prop></not></filter>

I have added two cmd files, one crash.cmd and one works.cmd.

The stack trace indicates the crash happens in the framework, so I will move this issue there.

The repro can be found here: https://github.com/nunit/nunit3-vs-adapter.issues/tree/master/Issue1146/NUnitFilterSample/src

[thomasbiel]

Is the filter XML you dumped as expected? I would like to contribute if I can, maybe you can give a hint where to start.

[OsirisTerje]

Yes, the filter is as expected.

You have to debug the framework. You can either use NUnitLite as the runner, and set up the same filter there, or you can create a new sln with both the framework and the adapter and set the adapter for debugging. The command for that is here: https://docs.nunit.org/articles/vs-test-adapter/Debugging.html

The call chain will also pass through the engine, but you can ignore that one. When it stops in the adapter, then you set further breakpoints in the framework.

[thomasbiel]

first analysis shows that TNode.FromXml produces an invalid output for the XML you dumped:

which leads to a follow-up error since NotFilter does not expect to contain an empty node list.
Are you sure the XML should look like (1) and not like (2)?

[BrightLight]

We tried to update from NUnit 3.16.0 3.14.0 to 4.0.1 today and encountered the same problem.

everything works fine until there is a test marked with [Explicit] along with non-marked tests in a fixture.

The exception happens also if there is a whole fixture that is marked as [Explicit].
I checked our project for use of [Explicit] and found two locations. One was on a test method within within a fixture (which had other, non-explicit tests). The other location was on a fixture itself.

[TestFixture]
public sealed class Foo
{
  [Test]
  public void Test1()  { }

  [Test]
  [Explicit]
  public void Test2()  { }
}

[Explicit]
public sealed class Bar
{
  [Test]
  public void Test1()  { }
}

I had to remove both to get rid of the exception.

[OsirisTerje]

@BrightLight 3.16 ? There is no NUNit 3.16. Do you mean the console ?

[BrightLight]

@OsirisTerje I'm sorry. I meant NUnit 3.14.0. And NUnit3TestAdapter 4.5.0.

[OsirisTerje]

@BrightLight Without any filters, it works as it should, ref https://github.com/nunit/nunit.issues/tree/main/Issue4589

What kind of filters do you use? In VS or on cmd line with dotnet test ?

[BrightLight]

@OsirisTerje we use dotnet test. This is the command line from our build job:

dotnet test %PROJECT_HOME_NET%\bin\%OUTDIR%\net6.0-windows\MyAssemblyTests.dll --filter "TestCategory!=Benchmark & TestCategory!=DbSchemaDependent & TestCategory!=Integration & TestCategory!=UI" --blame-hang-timeout 10min --logger "trx;LogFilePrefix=NUnit-net6.0-windows" --results-directory "%PROJECT_HOME_NET%\bin\%OUTDIR%" 

This particular project defines 3854 tests. The tests of the two fixtures where [Explicit] is used don't define a [Category] attribute, that's why I didn't add it to my example code. But there's one testfixture in the project that uses the "Integration" category (which therefore is excluded by the command line filter). But this fixture is not the one that has [Explicit] tests.

Sorry if I caused confusion. I just wanted to underline that we can reproduce the originally described issue, but also that the issue also occurs if whole fixtures are marked as explicit and the project has (other) tests which are excluded by a filter.

[OsirisTerje]

@thomasbiel Have you had any success in debugging this further?

[thomasbiel]

No, before doing anything I need an answer to my question from #4589 (comment)

[OsirisTerje]

Sorry, I missed that question.
The (1) looks wrong. The second looks more correct, but given we can have also have or statements, it might be an outer and that is missing ?

[miky-quadient]

We have also encountered this problem during upgrade of nUnit to v4 in two projects. I tried removing the use of the explicit attribute as a workaround. In one project it helped, in the other it didn't.

[OsirisTerje]

@miky-quadient What are your filter statements?

[miky-quadient]

@OsirisTerje
dotnet test Packages.sln --filter TestCategory!=LinuxUnstable&TestCategory!=AzureEmulator&TestCategory!=DevopsUnstable

[maedula]

Same here. We want to transition our project to net8.0 and upgraded to NUnit v4 but are blocked by this bug. So we would also greatly appreciate a bugfix for this issue.

[thomasbiel]

@maedula upgrading to net8 only requires the update of NUnitTestAdapter, which works fine with NUnit 3.4, although this is (as in my case) not the preferred solution.
Another workaround is to create your own ExplicitAttribute with the least nested namespace shared with all your test classes, so it gets found before NUnit.Framework.ExplicitAttribute, e.g.

using System;
using NUnit.Framework;

namespace YourOwnNamespace; 

[AttributeUsage(AttributeTargets.Class | AttributeTargets.Method | AttributeTargets.Assembly, AllowMultiple = true)]
public class ExplicitAttribute : CategoryAttribute
{
    public ExplicitAttribute(string description = null) : base(Category) { }

    public const string Category = "Explicit";
}

That allows filtering explicit tests via category without having to edit all appearances of Explicit on tests.

[stevenaw]

@OsirisTerje Does this look similar to #3979 to you given the interplay with Explicit?

If so I'm wondering if we should close #3979 given this issue has momentum

[OsirisTerje]

@stevenaw Yes. Makes sense .

[stevenaw]

Done (duplicate closed)!

[mbenedykconfigit]

Are there any updates regarding to this issue? Will it be fixed?

[OsirisTerje]

@mbenedykconfigit Yes it will be fixed.

[SamWilliamsGS]

We're still seeing this problem on .NET 8 with NUnit 4.1.0. Is a fix on the roadmap? 🙂 Any test file with Explicit in there is pretty borked in different ways 😅