Reduce not generating unique enum values when database is not consistent

[loraderon]

I'm not sure if this should work or not, I would expect it to work since I'm grouping on an enum. But I also understand that the database has no concept of enums during indexing.

It's not possible to fix it with Enum.Parse either:

group result by new { Day = Enum.Parse<DayOfWeek>(result.Day.ToString()) } into g

Because then I get:

IndexCompilationException: Cannot extract field name from: Day = Enum.Parse<string>(result.Day.ToString()), IndexDefinitionProperty='Reduce', ProblematicText='results.GroupBy(result => new {
    Day = Enum.Parse < string > (result.Day.ToString())

---

Test case and failure

Assert.Single() Failure: The collection contained 3 items
Collection: [
  Result { Day = Monday, Values = ["monday-untouched"] }, 
  Result { Day = Monday, Values = ["monday-num"] }, 
  Result { Day = Monday, Values = ["monday-lower"] }
]

public class EnumReduceTests : RavenTestDriver
{
    [Fact]
    public void CanReduceEnum()
    {
        using var store = GetDocumentStore();
        store.ExecuteIndex(new FooIndex());

        using (var session = store.OpenSession())
        {
            var untouched = new Foo
            {
                Day = DayOfWeek.Monday,
                Value = "monday-untouched",
            };
            var saveEnumAsInteger = new Foo
            {
                Day = DayOfWeek.Monday,
                Value = "monday-num",
            };
            var lower = new Foo
            {
                Day = DayOfWeek.Monday,
                Value = "monday-lower",
            };
            session.Store(untouched);
            session.Store(saveEnumAsInteger);
            session.Store(lower);

            session.SaveChanges();

            // simulate data inconsistencies, like when migrating from SaveEnumAsInteger = true to SaveEnumAsInteger = false
            session.Advanced.Defer(new PatchCommandData(saveEnumAsInteger.Id, null, new() { Script = @"this.Day = 1" }));
            session.Advanced.Defer(new PatchCommandData(lower.Id, null, new() { Script = @"this.Day = 'monday'" }));

            session.SaveChanges();
        }
        
        WaitForIndexing(store);

        using (var session = store.OpenSession())
        {
            var days = session
                .Query<FooIndex.Result, FooIndex>()
                .ToList();

            Assert.Single(days);
        }
    }

    public class Foo
    {
        public string Id { get; set; }

        public DayOfWeek Day { get; set; }

        public string Value { get; set; }
    }

    public class FooIndex : AbstractIndexCreationTask<Foo, FooIndex.Result>
    {
        public FooIndex()
        {
            Map = foos =>
                from f in foos
                select new Result
                {
                    Day = f.Day,
                    Values = new[] { f.Value },
                };

            Reduce = results =>
                from result in results
                group result by new { result.Day } into g
                select new Result
                {
                    Day = g.Key.Day,
                    Values = g.SelectMany(x => x.Values).ToArray()
                };
        }

        public class Result
        {
            public DayOfWeek Day { get; set; }
            public string[] Values { get; set; }
        }
    }
}

[ayende]

Your issue is that on the server side, this is not an enum, but a string.
And when you are reducing on that, you are using case sensitive match.
When you read it back, the Monday and monday are being treated as the same value. But they are not reduced as such.

You should either ensure that you use consistent casing or use:

                group result by new {Day = result.Day.ToLower() } into g

[loraderon]

I think that I shouldn't have included the lowercase version because that wasn't my actual issue.

The issue I'm having is when the enum have previously been stored as an integer (SaveEnumsAsIntegers = true) and I want to migrate away from that without patching millions of documents.

[ayende]

Okay, https://xyproblem.info/

If you want to mix and match the values that are both strings and numbers, without patching, there are a few ways.

That said, I would recommend just patching them. There are options to limit the number of operations / sec:

So it wouldn't overload too much, give it a few hours, and it is done and the system is clean.

That said, remember that your types do not exist on the server side. So you'll need to do something like:

// *Additional Source*
public static class IndexHelper
{
    public static string[] DaysOfWeek = new string[] { "Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday" };
    public static string EnsureEnum(object value)
    {
        if (value is string s)
        {
            return s;
        }
        return DaysOfWeek[(int)value];
    }
}

As Additional Sources and then

                group result by new {Day = IndexHelper.EnsureEnum(result.Day) } into g

[loraderon]

Okay, https://xyproblem.info/

😆 My bad.

---

Thanks for the AdditionalSources sample but I have problems with getting it to work.

I'm getting a similar error as with Enum.Parse, looks like it cannot find the additional source maybe?:

Raven.Client.Exceptions.Documents.Compilation.IndexCompilationException : IndexCompilationException: Cannot extract field name from: Day = IndexHelper.EnsureEnum(result.Day), IndexDefinitionProperty='Reduce', ProblematicText='results.GroupBy(result => new {
    Day = IndexHelper.EnsureEnum(result.Day)
}).Select(g => new {
    Day = g.Key.Day,
    Values = Enumerable.ToArray(g.SelectMany(x => x.Values))
})' ---> System.NotSupportedException: Cannot extract field name from: Day = IndexHelper.EnsureEnum(result.Day)

using System;
using Raven.Client.Documents.Commands.Batches;
using Raven.Client.Documents.Indexes;
using Raven.TestDriver;
using Xunit;

namespace RavenEnumReduce;

public class EnumReduceTests : RavenTestDriver
{
    [Fact]
    public void CanReduceEnum()
    {
        using var store = GetDocumentStore();

        store.ExecuteIndex(new FooIndex());

        using (var session = store.OpenSession())
        {
            var untouched = new Foo
            {
                Day = DayOfWeek.Monday,
                Value = "monday-untouched",
            };
            var saveEnumAsInteger = new Foo
            {
                Day = DayOfWeek.Monday,
                Value = "monday-num",
            };
            session.Store(untouched);
            session.Store(saveEnumAsInteger);

            session.SaveChanges();

            // simulate data inconsistencies, like when migrating from SaveEnumAsInteger = true to SaveEnumAsInteger = false
            session.Advanced.Defer(new PatchCommandData(saveEnumAsInteger.Id, null, new() { Script = @"this.Day = 1" }));

            session.SaveChanges();
        }

        WaitForIndexing(store);

        using (var session = store.OpenSession())
        {
            var days = session
                .Query<FooIndex.Result, FooIndex>()
                .ToList();

            Assert.Single(days);
        }
    }

    public class Foo
    {
        public string Id { get; set; }

        public DayOfWeek Day { get; set; }

        public string Value { get; set; }
    }

    public class FooIndex : AbstractIndexCreationTask<Foo, FooIndex.Result>
    {
        public FooIndex()
        {
            Map = foos =>
                from f in foos
                select new Result
                {
                    Day = f.Day,
                    Values = new[] { f.Value },
                };

            Reduce = results =>
                from result in results
                group result by new { Day = IndexHelper.EnsureEnum(result.Day) } into g
                select new Result
                {
                    Day = g.Key.Day,
                    Values = g.SelectMany(x => x.Values).ToArray()
                };
            
            AdditionalSources = new()
            {
                {
                    "RavenEnumReduce",
                    @"
                    public static class IndexHelper 
                    {
                        public static string[] DaysOfWeek = new string[] { ""Sunday"", ""Monday"", ""Tuesday"", ""Wednesday"", ""Thursday"", ""Friday"", ""Saturday"" };
                        public static string EnsureEnum(object value)
                        {
                            if (value is string s)
                            {
                                return s;
                            }
                            return DaysOfWeek[(int)value];
                        }
                    }
                    "
                }
            };
        }

        public class Result
        {
            public DayOfWeek Day { get; set; }
            public string[] Values { get; set; }
        }

        public static class IndexHelper
        {
            public static DayOfWeek EnsureEnum(object value) => throw new NotImplementedException();
        }
    }
}

[arekpalinski]

The exception System.NotSupportedException: Cannot extract field name from: Day = IndexHelper.EnsureEnum(result.Day) is an internal issue in the RavenDB's index compiler. We'll check that (https://issues.hibernatingrhinos.com/issue/RavenDB-22317/Index-compiler-cannot-extract-Group-By-field-name).

You can use IndexHelper.EnsureEnum() in the map function instead of the reduce to workaround that. Also I had to modify the extension method to:

public static string EnsureEnum(object value)
{
	if (value is string s)
	{
		return s;
	}

	if (value is Sparrow.Json.LazyStringValue lsv)
	{
		return lsv.ToString();
	}

	return DaysOfWeek[(int)value];
}

alternatively you could also try something like:

public static string EnsureEnum(object value)
{
	try
	{
		var enumValue = System.Convert.ChangeType(value, typeof(int));
		return DaysOfWeek[(int)enumValue];
	}
	catch
	{
	}
	
	return value.ToString();
}

The working index definition is:

public FooIndex()
{
    Map = foos =>
        from f in foos
        select new Result
        {
            Day = IndexHelper.EnsureEnum(f.Day),
            Values = new[] { f.Value },
        };

    Reduce = results =>
        from result in results
        group result by new { Day = result.Day } into g
        select new Result
        {
            Day = g.Key.Day,
            Values = g.SelectMany(x => x.Values).ToArray()
        };

    AdditionalSources = new()
{
    {
        "RavenEnumReduce",
        @"
        public static class IndexHelper 
        {
            public static string[] DaysOfWeek = new string[] { ""Sunday"", ""Monday"", ""Tuesday"", ""Wednesday"", ""Thursday"", ""Friday"", ""Saturday"" };
            public static string EnsureEnum(object value)
            {
                if (value is string s)
                {
                    return s;
                }

                if (value is Sparrow.Json.LazyStringValue lsv)
                {
                    return lsv.ToString();
                }

                return DaysOfWeek[(int)value];
            }
        }
        "
    }
};
}

[loraderon]

Thank you!