Query: review (JSON) materializer code for queries with NoTrackingWithIdentityResolution

[maumar]

JSON specific materializer code treats NoTrackingWithIdentityResolution as if it was a regular NoTracking query. However, EntityMaterializerInjectingExpressionVisitor mostly pays attention to the presence of change tracker (i.e. materializer template code that is generated is roughly the same between Tracking and NoTrackingWithIdentityResolution.

This results in discrepancy, that can lead to bugs (e.g. #33073)
On top of that, we block some scenarios for Tracking or NoTracking queries, e.g. you can't project owned type without it's owner in the tracking query. But what should be the behavior for NoTrackingWithIdentityResolution? We should look at all those cases as well.

[maumar]

issues to investigate:

Json_branch_collection_distinct_and_other_collection - NRE
Json_collection_distinct_in_projection - NRE
Json_collection_filter_in_projection - NRE
Json_collection_SelectMany - unable to track
Json_collection_skip_take_in_projection - NRE
Json_multiple_collection_projections - NRE
Json_nested_collection_anonymous_projection_in_projection - invalid token
Json_nested_collection_filter_in_projection - NRE
Json_nested_collection_SelectMany - unable to track
Json_projection_deduplication_with_collection_in_original_and_collection_indexer_in_target - invalid token
Json_projection_deduplication_with_collection_indexer_in_target - invalid token

Basic_json_projection_owner_entity_duplicated_NoTracking - invalid token
Basic_json_projection_owner_entity_duplicated_NoTrackingWithIdentityResolution - invalid token
Basic_json_projection_owner_entity_twice_NoTracking - invalid token
Basic_json_projection_owner_entity_twice_NoTrackingWithIdentityResolution - invalid token
Custom_naming_projection_everything - invalid token
Json_collection_after_collection_index_in_projection_using_constant_when_owner_is_present - invalid token
Json_collection_after_collection_index_in_projection_using_parameter_when_owner_is_present - invalid token
Json_projection_deduplication_with_collection_indexer_in_original - invalid token
Json_projection_with_deduplication - invalid token
Json_with_projection_of_mix_of_json_collections_json_references_and_entity_collection - invalid token
Left_join_json_entities_complex_projection - invalid token
Left_join_json_entities_complex_projection_json_being_inner - invalid token
Project_json_entity_FirstOrDefault_subquery_deduplication - invalid token
Project_json_entity_FirstOrDefault_subquery_deduplication_and_outer_reference - invalid token

[maumar]

Json_branch_collection_distinct_and_other_collection

ss.Set<JsonEntityBasic>()
  .OrderBy(x => x.Id)
  .Select(x => new
  {
    First = x.OwnedReferenceRoot.OwnedCollectionBranch.Distinct().ToList(), 
    Second = x.EntityCollection.ToList()
  })

Reason for NRE exceptions is that in InjectEntityMaterializers (specifically in ProcessEntityShaper) we build a piece of code which will try to retrieve entity from the state manager based on key.

                expressions.Add(
                    Assign(
                        entryVariable,
                        Call(
                            QueryCompilationContext.QueryContextParameter,
                            TryGetEntryMethodInfo,
                            Constant(primaryKey),
                            NewArrayInit(
                                typeof(object),
                                primaryKey.Properties
                                    .Select(
                                        p => valueBufferExpression.CreateValueBufferReadValueExpression(
                                            typeof(object),
                                            p.GetIndex(),
                                            p))),
                            Constant(!shaper.IsNullable),
                            hasNullKeyVariable)));

we loop through primary key properties and try to extract the values to get the key. This works fine for Owned types, but not for JSON when collections are involved. For JSON, we have synthesized key which doesn't correspond to anything in the database, we construct it during materialization, but the code here doesn't account for that - we try to find that value from the reader, we don't so we put null value instead. This in turn marks the key as nullKey and as a result entire entity (that should be there) is not being materialized, but the null value is returned. If there is supposed to be another nested collection afterwards, NRE is thrown.

[maumar]

Example test case:

    [ConditionalFact]
    public void Test_NTWIR_owned_type()
    {
        using (var ctx = new MyContext())
        {
            ctx.Database.EnsureDeleted();
            ctx.Database.EnsureCreated();
            var e = new MyEntity
            {
                Owned = new OwnedRoot
                {
                    Number = 10,
                    Nested = new List<OwnedBranch>
                    {
                        //new OwnedBranch { Foo = "f1", InnerNested = new List<OwnedLeaf> { new OwnedLeaf { Bar = 55 } } },
                        //new OwnedBranch { Foo = "f2", InnerNested = new List<OwnedLeaf> { new OwnedLeaf { Bar = 266 }, new OwnedLeaf { Bar = 277 } } }

                        new OwnedBranch { Foo = "f1" },
                        new OwnedBranch { Foo = "f2" }
                    }
                }
            };
            ctx.Entities.Add(e);
            ctx.SaveChanges();
        }

        using (var ctx = new MyContext())
        {
            var result =  ctx.Entities
                .OrderBy(x => x.Id)
                .Select(
                    x => new
                    {
                        First = x.Owned.Nested.ToList(),
                    })
                .AsNoTrackingWithIdentityResolution().ToList();
        }
    }

    public class MyContext : DbContext
    {
        public DbSet<MyEntity> Entities { get; set; }


        protected override void OnModelCreating(ModelBuilder modelBuilder)
        {
            modelBuilder.Entity<MyEntity>().OwnsOne(x => x.Owned, b =>
            {
                b.ToJson();
                //b.OwnsMany(xx => xx.Nested, bb => bb.OwnsMany(xxx => xxx.InnerNested));
                b.OwnsMany(xx => xx.Nested);
            });
        }

        protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        {
            optionsBuilder.UseSqlServer(@"Server=(localdb)\mssqllocaldb;Database=Repro;Trusted_Connection=True;MultipleActiveResultSets=true");
        }
    }

    public class MyEntity
    {
        public int Id { get; set; }
        public string Name { get; set; }

        public OwnedRoot Owned { get; set; }
    }

    public class OwnedRoot
    {
        public int Number { get; set; }

        public List<OwnedBranch> Nested { get; set; }
    }

    public class OwnedBranch
    {
        public string Foo { get; set; }

        //public List<OwnedLeaf> InnerNested { get; set; }
    }

    //public class OwnedLeaf
    //{
    //    public int Bar { get; set; }
    //}

working shaper (regular owned):

(queryContext, dataReader, resultContext, resultCoordinator) => 
{
    resultContext.Values == null ? 
    {
        List<OwnedBranch> namelessParameter{0};
        namelessParameter{0} = ShaperProcessingExpressionVisitor.InitializeCollection<OwnedBranch, List<OwnedBranch>>(
            collectionId: 0, 
            queryContext: queryContext, 
            dbDataReader: dataReader, 
            resultCoordinator: resultCoordinator, 
            parentIdentifier: Func<QueryContext, DbDataReader, object[]>, 
            outerIdentifier: Func<QueryContext, DbDataReader, object[]>, 
            clrCollectionAccessor: null);
        return resultContext.Values = new object[]{ namelessParameter{0} };
    } : default(void);
    ShaperProcessingExpressionVisitor.PopulateCollection<List<OwnedBranch>, OwnedBranch, OwnedBranch>(
        collectionId: 0, 
        queryContext: queryContext, 
        dbDataReader: dataReader, 
        resultCoordinator: resultCoordinator, 
        parentIdentifier: (queryContext, dataReader) => new object[]{ (object)(int?)dataReader.GetInt32(0) }, 
        outerIdentifier: (queryContext, dataReader) => new object[]{ (object)(int?)dataReader.GetInt32(0) }, 
        selfIdentifier: (queryContext, dataReader) => new object[]
        { 
            (object)dataReader.IsDBNull(1) ? default(int?) : (int?)dataReader.GetInt32(1), 
            (object)dataReader.IsDBNull(2) ? default(int?) : (int?)dataReader.GetInt32(2) 
        }, 
        parentIdentifierValueComparers: List<ValueComparer> { DefaultValueComparer<int> }, 
        outerIdentifierValueComparers: List<ValueComparer> { DefaultValueComparer<int> }, 
        selfIdentifierValueComparers: List<ValueComparer> { DefaultValueComparer<int>, DefaultValueComparer<int> }, 
        innerShaper: (queryContext, dataReader, resultContext, resultCoordinator) => 
        {
            OwnedBranch namelessParameter{1};
            namelessParameter{1} = 
            {
                MaterializationContext materializationContext1;
                IEntityType entityType1;
                OwnedBranch instance1;
                InternalEntityEntry entry1;
                bool hasNullKey1;
                materializationContext1 = new MaterializationContext(
                    ValueBuffer, 
                    queryContext.Context
                );
                instance1 = default(OwnedBranch);
                entry1 = queryContext.TryGetEntry(
                    key: Key: OwnedBranch.OwnedRootMyEntityId, OwnedBranch.Id PK, 
                    keyValues: new object[]
                    { 
                        dataReader.IsDBNull(1) ? default(object) : (object)dataReader.GetInt32(1), 
                        dataReader.IsDBNull(2) ? default(object) : (object)dataReader.GetInt32(2) 
                    }, 
                    throwOnNullKey: False, 
                    hasNullKey: hasNullKey1);
                !(hasNullKey1) ? entry1 != default(InternalEntityEntry) ? 
                {
                    entityType1 = entry1.EntityType;
                    return instance1 = (OwnedBranch)entry1.Entity;
                } : 
                {
                    ISnapshot shadowSnapshot1;
                    shadowSnapshot1 = Snapshot;
                    entityType1 = EntityType: OwnedBranch Owned;
                    instance1 = switch (entityType1)
                    {
                        case EntityType: OwnedBranch Owned: 
                            {
                                shadowSnapshot1 = (ISnapshot)new Snapshot<int, int>(
                                    dataReader.IsDBNull(1) ? default(int) : dataReader.GetInt32(1), 
                                    dataReader.IsDBNull(2) ? default(int) : dataReader.GetInt32(2)
                                );
                                return 
                                {
                                    OwnedBranch instance;
                                    instance = new OwnedBranch();
                                    instance.<Foo>k__BackingField = dataReader.IsDBNull(3) ? default(string) : dataReader.GetString(3);
                                    (instance is IInjectableService) ? ((IInjectableService)instance).Injected(
                                        context: materializationContext1.Context, 
                                        entity: instance, 
                                        bindingInfo: ParameterBindingInfo) : default(void);
                                    return instance;
                                }}
                        default: 
                            default(OwnedBranch)
                    }
                    ;
                    entry1 = entityType1 == default(IEntityType) ? default(InternalEntityEntry) : queryContext.StartTracking(
                        entityType: entityType1, 
                        entity: instance1, 
                        snapshot: shadowSnapshot1);
                    return instance1;
                } : default(void);
                return instance1;
            };
            return namelessParameter{1};
        });
    return IsTrue(resultCoordinator.ResultReady)
     ? new { First = (List<OwnedBranch>)(resultContext.Values[0]) } : default(<>f__AnonymousType21<List<OwnedBranch>>);
}

failing shaper (json)

(queryContext, dataReader, resultContext, resultCoordinator) => 
{
    resultContext.Values == null ? 
    {
        List<OwnedBranch> namelessParameter{0};
        namelessParameter{0} = ShaperProcessingExpressionVisitor.InitializeCollection<OwnedBranch, List<OwnedBranch>>(
            collectionId: 0, 
            queryContext: queryContext, 
            dbDataReader: dataReader, 
            resultCoordinator: resultCoordinator, 
            parentIdentifier: Func<QueryContext, DbDataReader, object[]>, 
            outerIdentifier: Func<QueryContext, DbDataReader, object[]>, 
            clrCollectionAccessor: null);
        return resultContext.Values = new object[]{ namelessParameter{0} };
    } : default(void);
    ShaperProcessingExpressionVisitor.PopulateCollection<List<OwnedBranch>, OwnedBranch, OwnedBranch>(
        collectionId: 0, 
        queryContext: queryContext, 
        dbDataReader: dataReader, 
        resultCoordinator: resultCoordinator, 
        parentIdentifier: (queryContext, dataReader) => new object[]{ (object)(int?)dataReader.GetInt32(0) }, 
        outerIdentifier: (queryContext, dataReader) => new object[]{ (object)(int?)dataReader.GetInt32(0) }, 
        selfIdentifier: (queryContext, dataReader) => new object[]{ dataReader.IsDBNull(3) ? default(string) : dataReader.GetString(3) }, 
        
		
		
		
		parentIdentifierValueComparers: List<ValueComparer> { DefaultValueComparer<int> }, 
        outerIdentifierValueComparers: List<ValueComparer> { DefaultValueComparer<int> }, 
        selfIdentifierValueComparers: List<ValueComparer> { DefaultValueComparer<string> }, 
        innerShaper: (queryContext, dataReader, resultContext, resultCoordinator) => 
        {
            OwnedBranch namelessParameter{1};
            namelessParameter{1} = 
            {
                MaterializationContext materializationContext1;
                IEntityType entityType1;
                OwnedBranch instance1;
                InternalEntityEntry entry1;
                bool hasNullKey1;
                materializationContext1 = new MaterializationContext(
                    ValueBuffer, 
                    queryContext.Context
                );
                instance1 = default(OwnedBranch);
                entry1 = queryContext.TryGetEntry(
                    key: Key: OwnedBranch.OwnedRootMyEntityId, OwnedBranch.Id PK, 
                    keyValues: new object[]
                    { 
                        dataReader.IsDBNull(1) ? default(object) : (object)dataReader.GetInt32(1), 
                        default(object) 
                    }, 
                    throwOnNullKey: False, 
                    hasNullKey: hasNullKey1);
                !(hasNullKey1) ? entry1 != default(InternalEntityEntry) ? 
                {
                    entityType1 = entry1.EntityType;
                    return instance1 = (OwnedBranch)entry1.Entity;
                } : 
                {
                    ISnapshot shadowSnapshot1;
                    shadowSnapshot1 = Snapshot;
                    entityType1 = EntityType: OwnedBranch Owned;
                    instance1 = switch (entityType1)
                    {
                        case EntityType: OwnedBranch Owned: 
                            {
                                shadowSnapshot1 = (ISnapshot)new Snapshot<int, int>(
                                    dataReader.IsDBNull(1) ? default(int) : dataReader.GetInt32(1), 
                                    default(int)
                                );
                                return 
                                {
                                    OwnedBranch instance;
                                    instance = new OwnedBranch();
                                    instance.<Foo>k__BackingField = dataReader.IsDBNull(2) ? default(string) : dataReader.GetString(2);
                                    (instance is IInjectableService) ? ((IInjectableService)instance).Injected(
                                        context: materializationContext1.Context, 
                                        entity: instance, 
                                        bindingInfo: ParameterBindingInfo) : default(void);
                                    return instance;
                                }}
                        default: 
                            default(OwnedBranch)
                    }
                    ;
                    entry1 = entityType1 == default(IEntityType) ? default(InternalEntityEntry) : queryContext.StartTracking(
                        entityType: entityType1, 
                        entity: instance1, 
                        snapshot: shadowSnapshot1);
                    return instance1;
                } : default(void);
                return instance1;
            };
            return namelessParameter{1};
        });
    return IsTrue(resultCoordinator.ResultReady)
     ? new { First = (List<OwnedBranch>)(resultContext.Values[0]) } : default(<>f__AnonymousType21<List<OwnedBranch>>);
}

[maumar]

covered by #33101, closing