Migrations blocked by: The type mapping for 'JsonElement' has not implemented code literal generation

[luca-esse]

Versions:
EF 8.0.2
Npgsql.EntityFrameworkCore.PostgreSQL 8.0.2
Microsoft.EntityFrameworkCore.Tools 8.0.2

We are facing an issue when try to add/rename/remove or change nullability of JsonElement fields after the first migration.

We are only able to generate an initial migration with JsonElements, after that, the add migration command fails with the following error:

PM> add-migration MySecondMigration
Build started...
Build succeeded.
System.NotSupportedException: The type mapping for 'JsonElement' has not implemented code literal generation.
   at Microsoft.EntityFrameworkCore.Storage.CoreTypeMapping.GenerateCodeLiteral(Object value)
   at Microsoft.EntityFrameworkCore.Design.Internal.CSharpHelper.UnknownLiteral(Object value)
   at Microsoft.EntityFrameworkCore.Migrations.Design.CSharpMigrationOperationGenerator.Generate(AddColumnOperation operation, IndentedStringBuilder builder)
   at System.Dynamic.UpdateDelegates.UpdateAndExecuteVoid3[T0,T1,T2](CallSite site, T0 arg0, T1 arg1, T2 arg2)
   at Microsoft.EntityFrameworkCore.Migrations.Design.CSharpMigrationOperationGenerator.Generate(String builderName, IReadOnlyList`1 operations, IndentedStringBuilder builder)
   at Microsoft.EntityFrameworkCore.Migrations.Design.CSharpMigrationsGenerator.GenerateMigration(String migrationNamespace, String migrationName, IReadOnlyList`1 upOperations, IReadOnlyList`1 downOperations)
   at Microsoft.EntityFrameworkCore.Migrations.Design.MigrationsScaffolder.ScaffoldMigration(String migrationName, String rootNamespace, String subNamespace, String language)
   at Microsoft.EntityFrameworkCore.Design.Internal.MigrationsOperations.AddMigration(String name, String outputDir, String contextType, String namespace)
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.AddMigrationImpl(String name, String outputDir, String contextType, String namespace)
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.AddMigration.<>c__DisplayClass0_0.<.ctor>b__0()
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.OperationBase.<>c__DisplayClass3_0`1.<Execute>b__0()
   at Microsoft.EntityFrameworkCore.Design.OperationExecutor.OperationBase.Execute(Action action)
The type mapping for 'JsonElement' has not implemented code literal generation.

Steps to reproduce:

1. With Visual Studio, create a Worker project
2. Add the following packages:

 <PackageReference Include="Npgsql.EntityFrameworkCore.PostgreSQL" Version="8.0.2" />
 <PackageReference Include="Microsoft.EntityFrameworkCore.Tools" Version="8.0.2">
  <PrivateAssets>all</PrivateAssets>
  <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
 </PackageReference>

4. Add the following classes:

internal class MyDbContext : DbContext
{
    public DbSet<MyEntity> MyEntities { get; set; }

    protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
    {
        base.OnConfiguring(optionsBuilder);

        optionsBuilder.UseNpgsql("MyConnectionString");
    }
}

internal class MyEntity
{
    public Guid Id { get; set; }
    public JsonElement MyJsonField1 { get; set; }

    // Uncomment this at step 5
    //public JsonElement MyJsonField2 { get; set; }
}

4. Execute the following command on the Package Manager Console: add-migration Initial
5. Uncomment the "MyJsonField2" property on the "MyEntity" class
6. Execute the following command on the Package Manager Console: add-migration MySecondMigration
7. The command will fail with the error specified in the title.

[luca-esse]

The issue seems to be related to this commit.

The JsonElement is now mapped to NpgsqlOwnedJsonTypeMapping but this class doesn't have the GenerateCodeLiteral method implemented and thus the NotSupportedException is raised from the RelationalTypeMapping base class.

I've opened a PR with the implementation for this method, taken from the original mapper

[Laurianti]

This is a significant regression that seriously affects the ability to use JsonElement.

It warrants a more thorough investigation to understand the full impact and find a viable solution.

dotnet/efcore#32192

[roji]

tl;dr to work around this in 8.0, configure the new jsonb column with an explicit default value as follows:

protected override void OnModelCreating(ModelBuilder modelBuilder)
{
    modelBuilder.Entity<Blog>().Property(b => b.JsonElement).HasDefaultValueSql("{}");
}

Alternatively, using JsonDocument instead of JsonElement should make the problems go away.

---

Note that adding a JsonElement property and generating a migration never properly worked (so I'm not sure there's an actual regression here - @Laurianti some details would be good). Using 7.0, the generated migration was:

migrationBuilder.AddColumn<JsonElement>(
    name: "JsonElement",
    table: "Blogs",
    type: "jsonb",
    nullable: false,
    defaultValue: System.Text.Json.JsonDocument.Parse("", new System.Text.Json.JsonDocumentOptions()).RootElement);

Note the invalid empty string argument to Parse; when trying to actually generate the SQL for this migration, one gets:

The input does not contain any JSON tokens. Expected the input to start with a valid JSON token, when isFinalBlock is true. LineNumber: 0 | BytePositionInLine: 0.

The workaround above with HasDefaultValueSql should take care of that problem as well.

---

Here's what's going on with JsonElement:

• The current state with JsonElement properties is hacky, in order to support the new EF owned JSON support; since EF expects the "owned" mapping when looking up JsonElement, we return that, but use NpgsqlJsonElementHackConvention to set the non-owned NpgsqlJsonTypeMapping on JsonElement properties. Remove lookup of JsonTypeMapping via JsonElement dotnet/efcore#32192 is about cleaning that up.
• However, when EF generates an AddColumn migration, CSharpMigrationOperationGenerator sees a non-null DefaultValue, and calls CSharpHelper.UnknownLiteral, which looks up the default type mapping for JsonElement; because of the hacky state of affairs, it gets back NpgsqlOwnedJsonTypeMapping (for owned JSON support) and not the actual column's type mapping (NpgsqlJsonTypeMapping), which has proper C# literal generation implemented.
• In short, this will go away once Remove lookup of JsonTypeMapping via JsonElement dotnet/efcore#32192 is done (/cc @maumar).

[luca-esse]

@roji thank you, i can confirm that setting the default value avoids this bug and it might work for us in the meantime.

We cannot use the JsonDocument since it's an IDisposable and it would be a real nightmare to manage correctly in our scenario.
For this reason the JsonElement is very very important for us.

[roji]

@luca-esse if my understanding is correct, not disposing a JsonDocument only results in memory not being pooled (and therefore increased GC pressure) - there's no leak or any effect that's more serious (see docs). So AFAIK, reading JsonDocument should be OK.

[luca-esse]

@roji you are right, I've missed that part. But still, not ideal since it would trigger missing disposal warnings from code analyzers.
We also heavily depend on it (tons of Json fields) so going full JsonDocument is not feasible ATM

[ProTip]

I seem to be hitting this issue moving away from a JsonElement column to JsonNode. I'll try the default workaround and then hand jam the migration/designer files.