Assert.Multiple has no message parameter

[jmartschinke]

Imagine the following example:

var variable0 = 0;
var variable1 = 1;

for (int i = 0; i < 5; i++)
{
    Assert.Multiple(() =>
    {
        Assert.That(variable0, Is.Zero,       "first assertion");
        Assert.That(variable1, Is.EqualTo(1), "second assertion");
    });

    // do something that can change the outcome
    variable0++;
    variable1++;
}

Often times you want to check a block of assertions at different times like in this example.

When you run this code, you get something like the following:

Multiple failures or warnings in test:
  1)   first assertion
  Expected: 0
  But was:  1
  <stacktrace>
  2)   second assertion
Expected: 1
But was:  0
<stacktrace>

Now you cannot distinguish in which iteration the Assert.Multiple failed. What you would need to do is to write something like

Assert.That(variable0, Is.Zero,       $"iteration {i}, first assertion");
Assert.That(variable1, Is.EqualTo(1), $"iteration {i}, second assertion");

Which would work, but you have to repeat at every assertion instead of writing it only once to the Assert.Multiple. You get the following output:

Multiple failures or warnings in test:
  1)   iteration 1, first assertion
  Expected: 0
  But was:  1
  <stacktrace>
  2)   iteration 1, second assertion
Expected: 1
But was:  0
<stacktrace>

It would be better if you can write the code like this:

Assert.Multiple(() =>
  {
      Assert.That(variable0, Is.Zero,       "first assertion");
      Assert.That(variable1, Is.EqualTo(1), "second assertion");
  }, $"iteration {i}");

And then get this output:

iteration 1
  Multiple failures or warnings in test:
  1)   first assertion
  Expected: 0
  But was:  1
  <stacktrace>
  2)   second assertion
Expected: 1
But was:  0
<stacktrace>

[stevenaw]

Thanks for your thorough proposal @jmartschinke I've tried reviewing the original issue to understand some of the design decisions there but I can't see a clear description why there is no existing message parameter.

I understand this feature would bring clarity for your case but I'm a little concerned it could cause confusion for others. Elsewhere in the framework the message parameter would replace the default assertion rather than supplement it. Now, I don't think replacing the default message is what we want here (it would hide understanding which of the contained assertions actually failed!) which could explain why it was omitted up front. Conversely, and more to your suggestion, if it were to be added as a "supplemental" message here I'd be afraid of it being confusing for other use cases as the same logical parameter then gets treated differently by different asserts in the framework.

That said, I think it's a nifty idea to be able to track and report on extra "context" within the test like this if we could generalize it 🙂

For a more immediate solution for you, if the intent is to write this as part of the test result, have you taken a look at using TestContext.WriteLine ? I haven't tested the below code but I'm curious if it works for you:

for (var i = 0; i < count; i++)
{
  TestContext.WriteLine($"iteration {i}");

  Assert.Multiple(() => {
    // Your asserts here
  });
}

[jmartschinke]

Thank you for your help, this is indeed very helpful to know!
I did however just create a helper class for this:

/// <summary>
/// Wraps code containing a series of assertions, which should all
/// be executed, even if they fail. Failed results are saved and
/// reported at the end of the code block.
/// </summary>
/// <param name="testDelegate">A TestDelegate to be executed in Multiple Assertion mode.</param>
/// <param name="multipleAssertBlockDescription">A description for the multiple assert block</param>
public static void Multiple(TestDelegate testDelegate, string multipleAssertBlockDescription)
{
    try
    {
        Assert.Multiple(testDelegate);
    }
    catch (MultipleAssertException ex)
    {
        var msg = ex.Message;

        const string multipleFailuresText = "Multiple failures or warnings in test:";

        if (msg.StartsWith(multipleFailuresText))
        {
            msg = $"Multiple failures or warnings in multiple assert block {multipleAssertBlockDescription}{msg[multipleFailuresText.Length..]}";
        }
        else
        {
            msg = $"Failure or warning in multiple assert block {multipleAssertBlockDescription}{Environment.NewLine}{msg}";
        }

        ex.GetType()
          .GetField("_message", BindingFlags.NonPublic | BindingFlags.Instance)
          .SetValue(ex, msg);

        throw ex;
    }
}

[OsirisTerje]

Interesting. Can you show (e.g. some screenshots) how this looks in the Visual Studio Test Explorer and in dotnet test ?

[jmartschinke]

Of course :)
These are some example tests:

[Test]
public void TestMultipleFailures()
{
    Asserts.Multiple(() =>
    {
        Assert.That(true, Is.True);
        Assert.That(true, Is.True);
    }, "1");


    Asserts.Multiple(() =>
    {
        Assert.That(true, Is.False);
        Assert.That(true, Is.False);
    }, "2");
}

[Test]
public void TestSingleFailure()
{
    Asserts.Multiple(() =>
    {
        Assert.That(true, Is.True);
        Assert.That(true, Is.True);
    }, "1");


    Asserts.Multiple(() =>
    {
        Assert.That(true, Is.True);
        Assert.That(true, Is.False);
    }, "2");
}

Output of the Visual Studio Test Explorer:

Output of dotnet test:
Failed TestMultipleFailures [20 ms]
Error Message:
Multiple failures or warnings in multiple assert block 2

1. Assert.That(true, Is.False)
   Expected: False
   But was: True

2. Assert.That(true, Is.False)
   Expected: False
   But was: True
   <stacktrace>

Failed TestSingleFailure [< 1 ms]
Error Message:
Failure or warning in multiple assert block 2
Assert.That(true, Is.False)
Expected: False
But was: True
<stacktrace>

[kmoltke]

Thank you for your help, this is indeed very helpful to know! I did however just create a helper class for this:

/// <summary>
/// Wraps code containing a series of assertions, which should all
/// be executed, even if they fail. Failed results are saved and
/// reported at the end of the code block.
/// </summary>
/// <param name="testDelegate">A TestDelegate to be executed in Multiple Assertion mode.</param>
/// <param name="multipleAssertBlockDescription">A description for the multiple assert block</param>
public static void Multiple(TestDelegate testDelegate, string multipleAssertBlockDescription)
{
    try
    {
        Assert.Multiple(testDelegate);
    }
    catch (MultipleAssertException ex)
    {
        var msg = ex.Message;

        const string multipleFailuresText = "Multiple failures or warnings in test:";

        if (msg.StartsWith(multipleFailuresText))
        {
            msg = $"Multiple failures or warnings in multiple assert block {multipleAssertBlockDescription}{msg[multipleFailuresText.Length..]}";
        }
        else
        {
            msg = $"Failure or warning in multiple assert block {multipleAssertBlockDescription}{Environment.NewLine}{msg}";
        }

        ex.GetType()
          .GetField("_message", BindingFlags.NonPublic | BindingFlags.Instance)
          .SetValue(ex, msg);

        throw ex;
    }
}

Very low level question: Where did you implement your helper method? Did you extend anything? If so, what? I see the method but not the helper class :)

[jmartschinke]

Very low level question: Where did you implement your helper method? Did you extend anything? If so, what? I see the method but not the helper class :)

I just added a static class named Asserts, so i can write Asserts.Multiple (instead of Assert.Multiple) to avoid name issues.