MemberAutoData only uses first entry from the supplied enumerable

[nirwall]

In AutoFixture.xUnit2, the MemberAutoData attribute only causes one test to be generated regardless of the size of the supplied data. You can see this in the test AutoFixture.Xunit2.UnitTest.Scenario.MemberAutoDataSuppliesDataSpecimens.

A similar test with MemberData and a member supplying a similar but "full" enumerable will generate two test cases:

[Theory, MemberData(nameof(StringData))]
public void MemberDataSuppliesDataSpecimens(string s1, string s2, MyClass myClass)
{
    Assert.Equal("foo", s1);
    Assert.NotNull(s2);
    Assert.NotNull(myClass);
}

public static IEnumerable<object[]> StringData
{
    get
    {
        yield return new object[] { "foo", "bar", new MyClass() };
        yield return new object[] { "foo", "baz", new MyClass() };
    }
}

[Ergamon]

The problem is that the implementation uses the CompositeDataAttribute base class. This one tries to merge the data from MemberDataAttribute and AutoDataAttribute, but the later always returns one row. So in your example MemberDataAttribute returns 2 rows, AutoDataAttribute only 1. The code merges this to the smallest number found, so only 1 row is generated.

I tried my best understanding of the AutoFixture source code and generated a (hopefully) working implementation:

  using System;
  using System.Collections.Generic;
  using System.Linq;
  using System.Reflection;
  using System.Threading;
  using AutoFixture;
  using AutoFixture.Kernel;
  using AutoFixture.Xunit2;
  using Xunit;
  using Xunit.Sdk;

  public class MemberAutoDataAttribute : DataAttribute
  {
    private readonly Lazy<IFixture> fixture;
    private readonly MemberDataAttribute memberDataAttribute;

    public MemberAutoDataAttribute(string memberName, params object[] parameters)
      : this(memberName, parameters, () => new Fixture())
    {
    }

    protected MemberAutoDataAttribute(string memberName, object[] parameters, Func<IFixture> fixtureFactory)
    {
      if (fixtureFactory == null)
      {
        throw new ArgumentNullException(nameof(fixtureFactory));
      }
      
      memberDataAttribute = new MemberDataAttribute(memberName, parameters);
      fixture = new Lazy<IFixture>(fixtureFactory, LazyThreadSafetyMode.PublicationOnly);
    }

    public override IEnumerable<object[]> GetData(MethodInfo testMethod)
    {
      if (testMethod == null)
      {
        throw new ArgumentNullException(nameof(testMethod));
      }

      var memberData = memberDataAttribute.GetData(testMethod);

      using (var enumerator = memberData.GetEnumerator())
      {
        if (enumerator.MoveNext())
        {
          var specimens = GetSpecimens(testMethod.GetParameters(), enumerator.Current.Length).ToArray();

          do
          {
            yield return enumerator.Current.Concat(specimens).ToArray();
          } while (enumerator.MoveNext());
        }
      }
    }

    private IEnumerable<object> GetSpecimens(IEnumerable<ParameterInfo> parameters, int skip)
    {
      foreach (var parameter in parameters.Skip(skip))
      {
        CustomizeFixture(parameter);

        yield return Resolve(parameter);
      }
    }

    private void CustomizeFixture(ParameterInfo p)
    {
      var customizeAttributes = p.GetCustomAttributes()
          .OfType<IParameterCustomizationSource>()
          .OrderBy(x => x, new CustomizeAttributeComparer());

      foreach (var ca in customizeAttributes)
      {
        var c = ca.GetCustomization(p);
        fixture.Value.Customize(c);
      }
    }

    private object Resolve(ParameterInfo p)
    {
      var context = new SpecimenContext(fixture.Value);

      return context.Resolve(p);
    }

    private class CustomizeAttributeComparer : Comparer<IParameterCustomizationSource>
    {
      public override int Compare(IParameterCustomizationSource x, IParameterCustomizationSource y)
      {
        var xfrozen = x is FrozenAttribute;
        var yfrozen = y is FrozenAttribute;

        if (xfrozen && !yfrozen)
        {
          return 1;
        }

        if (yfrozen && !xfrozen)
        {
          return -1;
        }

        return 0;
      }
    }
  }

So basically I iterate over the results from MemberDataAttribute while adding everytime the data from AutoDataAttribute.

The CustomizeFixture and the Resolve methods are copies from AutoDataAttribute.
The CustomizeAttributeComparer class is a direct copy from the source code as the original class is internal.

The protected constructor is there to provide the possibility to inherit, e.g. while testing I made my own MemberAutoNSubstituteDataAttribute.

[moikot]

@Ergamon Thank you for the MemberAutoDataAttribute. I'm also struggling with MemberAutoData.

[Ergamon]

You are welcome.

Meanwhile I use a modified version of this code in our production environment.
The main flaw is the CompositeDataAttribute. First it expects all combined attributes to return the same amount of lines, which is not always the case.
Then it lets all attributes calculate their values, regardless if needed or not. While this results not in a visible bug, it still feels wrong to create values never used.
So in our code I use basically this code in a slightly modified version as base class for ClassAutoData, InlineAutoData and MemberAutoData.

Be warned in my post here I made a small mistake and used the LINQ Union operator. It must be a call to Concat. I corrected my post now.

[moikot]

@Ergamon thank you for letting me know about Union! I also did some modifications but didn't notice the mistake.

[thomOrbelius]

@Ergamon thank you for the workaround, it really helped me out.

Also we created a ClassAutoDataAttribute as well, with the slight modification of using ClassDataAttribute instead of MemberDataAttribute.

[sanjay-soni]

@thomOrbelius It would be great help if you provide ClassAutoDataAttribute code?

[aivascu]

I've published a feature preview package 5.0.0-dataattributes0001 that fixes this issue. If anyone wishes to help please test the package, and provide feedback. I'll leave it listed until it will be merged into the main v5 branch.

[jcouturest]

@aivascu #1433 does resolve this issue.

Using 5.0.0-dataattributes0001:

Using 4.18.1:

[jcouturest]

IMO: This should be backported to 4.18 as well. Just my $0.02

[aivascu]

@jcouturest, thank you for providing feedback. If there's any bugs or performance issues, please let me know.

I'd love to just move the fix to v4, but I can't, since this is technically a breaking change, and the project has made a pledge to follow SemVer.

I could technically publish a MemberAutoData2Attribute, but that would open a whole new set of problems.

I should have another window of free time in about a week. I hope to merge several features into the v5 branch and maybe publish a RC version.

[jbentus]

@aivascu what about reopen/reuse #1164, which was initially raised to fix this issue in the v4 build and then remove the latest commit that added the breaking change for v4? It may not be perfect for v5, but it might be good enough to fix this issue in the the v4 branch (cc in @jcouturest).