Leveraging generic attributes to get test-specific test data

[Kralizek]

Hi,

I just published a blog post where I describe a technique that aims at easing the pain of using the AutoData flow with multiple setups of the same type.

https://renatogolia.com/2023/11/29/leveraging-generic-attributes-test-specific-customizations-autofixture/

[GeraldLx]

I dont understand the point in using the generic attribute.

I use a custom version of the AutoDataAttribute now for years basically doing the same. The problem with generics is that you cannot have something like params. So if you want to use more than one IFixtureConfigurator, you have to add additional versions of the type.

So your implementation looks like this

[Theory]
[AutoData<MyConfigurator>]
public void Test(Foo foo)
{

while I use

[Theory]
[AutoData]
[MyConfigurator]
public void Test(Foo foo)
{

The advantage of the seperate attribute is that you can easily go one step further. These attributes I allow to be put not only on methods, but also on classes and assemblies.
By doing this you can use a configuration for a a set of tests without having to repeat yourself over and over.

[Kralizek]

How do you compose multiple configurations with multiple attributes on the same IFixture?

[GeraldLx]

As said I have re-written some of the basic AutoFixture xUnit classes to fullfill this demand.

I use my own version of the AutoDataAttribute as a starting point.

  /// <summary>
  /// Base class for combining a <see cref="DataAttribute"/> with AutoFixture.
  /// </summary>
  [EditorBrowsable(EditorBrowsableState.Never)]
  public abstract class CompositeAutoDataAttribute : DataAttribute
  {
    private readonly DataAttribute _dataAttribute;
    private readonly Lazy<IFixture> _fixture;

    /// <summary>
    /// Initializes a new instance of the <see cref="CompositeAutoDataAttribute"/> class.
    /// </summary>
    /// <param name="dataAttribute">The data attribute to be combined.</param>
    /// <param name="fixtureFactory">A factory method to create the fixture.</param>
    protected CompositeAutoDataAttribute(
      DataAttribute dataAttribute,
      Func<IFixture> fixtureFactory)
    {
      _dataAttribute = EnsureArg.IsNotNull(
        dataAttribute,
        nameof(dataAttribute));

      RepeatCount = -1;

      _fixture = new Lazy<IFixture>(() => CreateCustomizedFixture(fixtureFactory), LazyThreadSafetyMode.PublicationOnly);
    }

    /// <summary>
    /// Gets or sets the behavior for delegates.
    /// </summary>
    public int RepeatCount { get; set; }

    /// <summary>
    /// Gets or sets a value indicating whether delegates create always a new return value.
    /// </summary>
    public bool UniqueReturnValuesForDelegates { get; set; }

    /// <summary>
    /// Gets the combined data.
    /// </summary>
    /// <param name="testMethod">The test method.</param>
    /// <returns>The combined data as enumerable.</returns>
    public override IEnumerable<object[]> GetData(MethodInfo testMethod)
    {
      EnsureArg.IsNotNull(testMethod, nameof(testMethod));

      CustomizeFixture(testMethod);

      var data = _dataAttribute.GetData(testMethod);

      using (var enumerator = data.GetEnumerator())
      {
        if (enumerator.MoveNext())
        {
          var specimens = GetSpecimens(
            testMethod.GetParameters(),
            enumerator.Current.Length)
            .ToArray();

          do
          {
            yield return enumerator.Current.Concat(specimens).ToArray();
          }
          while (enumerator.MoveNext());
        }
      }
    }

    private static IEnumerable<ICustomization> GetCustomizations(MethodInfo testMethod)
    {
      foreach (var attribute in testMethod.DeclaringType.Assembly.GetCustomAttributes<FixtureCustomizationAttribute>())
      {
        yield return attribute;
      }

      foreach (var attribute in testMethod.DeclaringType.GetCustomAttributes<FixtureCustomizationAttribute>())
      {
        yield return attribute;
      }

      foreach (var attribute in testMethod.GetCustomAttributes<FixtureCustomizationAttribute>())
      {
        yield return attribute;
      }
    }

    private IFixture CreateCustomizedFixture(Func<IFixture> fixtureFactory)
    {
      EnsureArg.IsNotNull(fixtureFactory, nameof(fixtureFactory));

      var fixture = fixtureFactory();

      if (RepeatCount >= 0)
      {
        fixture.RepeatCount = RepeatCount;
      }

      if (UniqueReturnValuesForDelegates)
      {
        var delegateBuilder = new DelegateWithUniqueReturnValueBuilder();
        fixture.Customizations.Insert(0, delegateBuilder);
      }

      return fixture;
    }

    private void CustomizeFixture(MethodInfo testMethod)
    {
      var customizations = GetCustomizations(testMethod);
      foreach (var customization in customizations)
      {
        customization.Customize(_fixture.Value);
      }
    }

    private void CustomizeFixture(ParameterInfo p)
    {
      var customizeAttributes = p.GetCustomAttributes()
        .OfType<IParameterCustomizationSource>()
        .OrderBy(
          x => x,
          new CustomizeAttributeComparer());

      foreach (var ca in customizeAttributes)
      {
        var c = ca.GetCustomization(p);
        _fixture.Value.Customize(c);
      }
    }

    private IEnumerable<object> GetSpecimens(
      IEnumerable<ParameterInfo> parameters,
      int skip)
    {
      foreach (var parameter in parameters.Skip(skip))
      {
        CustomizeFixture(parameter);

        yield return Resolve(parameter);
      }
    }

    private object Resolve(ParameterInfo p)
    {
      var context = new SpecimenContext(_fixture.Value);

      return context.Resolve(p);
    }

    private class CustomizeAttributeComparer : Comparer<IParameterCustomizationSource>
    {
      public override int Compare(
        IParameterCustomizationSource x,
        IParameterCustomizationSource y)
      {
        var xfrozen = x is FrozenAttribute;
        var yfrozen = y is FrozenAttribute;

        if (xfrozen && !yfrozen)
        {
          return 1;
        }

        if (yfrozen && !xfrozen)
        {
          return -1;
        }

        return 0;
      }
    }
  }

As you can see it is basically a copy of the original AutoDataAttribute from AutoFixture.xUnit adding this handling of the customization attributes.

All other attributes are just passing the xUnit attribute on to this class. So I have MemberAutoDataAttribute, InlineAutoMemberAttribute and ClassAutoDataAttribute. Last but not least I have plain AutoDataAttribute passing on some NullDataAttribute.

From there on you can subclass further weith MemberAutoNSubstituteAutoDataAttribute, InlineAutoNSubstituteDataAttribute and so on.

Hope you get the idea.

Going deeper a discussion is probably the wrong place. If you are interested I can share the full code.

[Kralizek]

I appreciate a lot. It's very brilliant. But I'm not sure how applicable to NUnit.

[GeraldLx]

I never ever used NUnit so far, but it doesnt look any different.

I just took a quick look at the AutoDataAttribute in AutoFixture.NUnit3.

Basically it is the same. The constructor takes a factory method to create the IFixture. This method is called when the private Fixture property first time is read.

So everything of interest is happening in BuildFrom. This is basically the GetData of the AutoDataAttribute in AutoFixture.Xunit.

One of the parameters is of NUnit´s IMethodInfo, which has a MethodInfo property to get the .NET MethodInfo.

So you can do the same conversions in this class that I have done to the other one.

In the end all these test frameworks work the same. There is one point where the parameters of the test method are resolved.

The AutoFixture classes do the following at this point in time:

1. Create an instance of the IFixture interface (usually by the supplied factory method)
2. Iterate over all parameters and apply all necessary customizations while retrieving them
   (This is why the order is necessary for the FrozenAttribute)

All that my code modification does is that I apply the customization attribute based customizations after 1 and before 2.
I also do this in the order assembly first then class then method. Basically to have some overwrite capability. The more specific customization wins over the more generic. (Assembly is for all tests in this project, Class for all tests in this class and Method for this test only)

[Kralizek]

That is indeed a lot of work for someone landing on AutoFixture + NUnit/XUnit and the AutoData pattern.

Whilst less powerful and probably more verbose than your approach, the approach I show in my blog post feels easier and "just good enough".