How to retry failing tests?

[mmateevadeltek]

Hi, I have implemented a custom xunit test framework, inheriting from xunit classes and I am executing in parallel. For some reasons (using 3rd party libraries, that do not work well in parallel), there are few tests that fail on the first time they are executed, but if they are run again, they pass. My idea is to retry failing tests for example 3 times. This is the code that I used for that idea:

internal class ITTestMethodRunner : XunitTestMethodRunner
{
	private const int numberOfRetries = 3;
	private readonly IMessageSink diagnosticMessageSink;

	public ITTestMethodRunner(ITestMethod testMethod,
		IReflectionTypeInfo @class,
		IReflectionMethodInfo method,
		IEnumerable<IXunitTestCase> testCases,
		IMessageSink diagnosticMessageSink,
		IMessageBus messageBus,
		ExceptionAggregator aggregator,
		CancellationTokenSource cancellationTokenSource,
		object[] constructorArguments)
		: base(testMethod,
			  @class,
			  method,
			  testCases,
			  diagnosticMessageSink,
			  messageBus,
			  aggregator,
			  cancellationTokenSource,
			  constructorArguments)
	{
		this.diagnosticMessageSink = diagnosticMessageSink;
	}

	protected override async Task<RunSummary> RunTestCaseAsync(IXunitTestCase testCase)
	{
		var summary = new RunSummary();			
		var currentRetry = 0;

		while (currentRetry <= numberOfRetries)
		{
			try
			{
				var tempSummary = await base.RunTestCaseAsync(testCase);

				// If test passed, we can break the loop and return the summary
				if (tempSummary.Failed == 0)
				{
					summary = tempSummary;
					break;
				}
				// If it failed but we have retries left, we prepare for a retry
				else if (currentRetry < numberOfRetries)
				{
					diagnosticMessageSink.OnMessage(new DiagnosticMessage($"Retrying test {testCase.DisplayName} - attempt {currentRetry + 1} of {numberOfRetries}"));
				}
			}
			catch (Exception ex)
			{
				if (currentRetry >= numberOfRetries)
				{
					// If we've exhausted retries, we need to let the exception bubble up
					throw;
				}
				// Log the exception details if you wish to
			}
			currentRetry++;
		}

		return summary;
	}
}

I have a test method that is always failing on the first run, and succeeding thereafter, but when I run it, I get this summary:

========== Test run finished: 2 Tests (1 Passed, 1 Failed, 0 Skipped) run in 10.6 sec ==========

And the output of the test is this:

Test has multiple result outcomes
   1 Passed
   1 Failed

Results

    1)   Test 
      Duration: 162 ms

      Message: 
This test is designed to fail on the first run and pass thereafter.

      Stack Trace: 
IT_AccessTag.Test() line 30

      Standard Output: 
Database Db_1, Created
Run Mode: Default



    2)   Test 
      Duration: < 1 ms

      Standard Output: 
Database:  Db_1, Created
Run Mode: Default

Can you give me an advice how to do it properly? Thank you!

[bradwilson]

We have a sample that does this: https://github.com/xunit/samples.xunit/tree/main/RetryFactExample

What's happening here is that the message listener in the runner is seeing you run the test multiple times, because you're letting it see the result messages from both the failures and the successes. Our sample shows using a "delayed message bus" so that it hangs onto all the messages while the test runs, and will only ever send those messages along if the test passes, or if the fail count exceeds the limit.

[mmateevadeltek]

Thank you, @bradwilson !

Yes, I could get it working for a my custom Fact attribute, but I also use Theory, and I am having trouble to get this working. It seems I cannot use the same approach as I do for facts, because it need other parameters, right?

This is the error I get:
System.InvalidOperationException : The test method expected 1 parameter value, but 0 parameter values were provided.

And the code that I use is from your example:

public override async Task<RunSummary> RunAsync(IMessageSink diagnosticMessageSink,
												IMessageBus messageBus,
												object[] constructorArguments,
												ExceptionAggregator aggregator,
												CancellationTokenSource cancellationTokenSource)
{
	var runCount = 0;

	while (true)
	{
		// This is really the only tricky bit: we need to capture and delay messages (since those will
		// contain run status) until we know we've decided to accept the final result;
		var delayedMessageBus = new DelayedMessageBus(messageBus);

		var summary = await base.RunAsync(diagnosticMessageSink, delayedMessageBus, constructorArguments, aggregator, cancellationTokenSource);
		
		if (aggregator.HasExceptions || summary.Failed == 0 || ++runCount >= maxRetries)
		{
			delayedMessageBus.Dispose();  // Sends all the delayed messages
			return summary;
		}

		diagnosticMessageSink.OnMessage(new DiagnosticMessage("Execution of '{0}' failed (attempt #{1}), retrying...", DisplayName, runCount));
	}
}

[mmateevadeltek]

Hi, it is me again, I was able to do that for Theory as well.

The first thing is to change the constructor of the RetryTestCase to pass testMethodArguments as well, like this:

public RetryTestCase(int maxRetries,
	IMessageSink diagnosticMessageSink,
	TestMethodDisplay testMethodDisplay,
	TestMethodDisplayOptions defaultMethodDisplayOptions,
	ITestMethod testMethod,
	object[] testMethodArguments = null)
	: base(diagnosticMessageSink,
		  testMethodDisplay,
		  defaultMethodDisplayOptions,
		  testMethod,
		  testMethodArguments)
{
	this.maxRetries = maxRetries;
}

Second, I created my own TheoryDiscoverer in which I am overriding CreateTestCasesForDataRow where I can pass the arguments:

public class ITTheoryDiscoverer : TheoryDiscoverer
{
	public ITTheoryDiscoverer(IMessageSink diagnosticMessageSink)
		: base(diagnosticMessageSink)
	{ }

	protected override IEnumerable<IXunitTestCase> CreateTestCasesForDataRow(ITestFrameworkDiscoveryOptions discoveryOptions, 
		ITestMethod testMethod,
		IAttributeInfo theoryAttribute,
		object[] dataRow)
	{
		var testCases = base.CreateTestCasesForDataRow(discoveryOptions, testMethod, theoryAttribute, dataRow);

		foreach(var testCase in testCases)
		{
			var maxRetries = theoryAttribute.GetNamedArgument<int>("MaxRetries");

			if (maxRetries < 1)
				maxRetries = 3;

			yield return new RetryTestCase(maxRetries,
				DiagnosticMessageSink,
				discoveryOptions.MethodDisplayOrDefault(),
				discoveryOptions.MethodDisplayOptionsOrDefault(),
				testCase.TestMethod,
				testCase.TestMethodArguments);
		}
	}
}

[bradwilson]

I just updated the sample to include a theory version. The changes are fairly minimal. xunit/samples.xunit@8ad214a

You don't need to call base.CreateTestCasesForDataRow since you're going to be making your own, and the data you're extracting from them, you already have access to.