Binary incompatibility with NJsonSchema version 11.0.0

[NoahStolk]

Description

Our team is currently running into a problem where the current version of NJsonSchema referenced from the Confluent.SchemaRegistry.Serdes.Json library (10.6.3) is not binary compatible with the resolved version from our binaries (11.0.0). We require NJsonSchema version 11.0.0 due to having a dependency on NSwag.AspNetCore version 14.0.0, and we require NSwag.AspNetCore to be on version 14.0.0 because that is the only version compatible with .NET 8.

NSwag.AspNetCore 14.0.0 is requesting NJsonSchema 11.0.0 from NuGet, which is the highest version and therefore ends up being resolved. However, since Confluent.SchemaRegistry.Serdes.Json is requesting NJsonSchema 10.6.3, we are getting runtime errors such as:

System.MissingMethodException: Method not found: 'Newtonsoft.Json.JsonSerializerSettings NJsonSchema.Generation.JsonSchemaGeneratorSettings.get_ActualSerializerSettings()'.
   at Confluent.SchemaRegistry.Serdes.JsonSerializer`1.SerializeAsync(T value, SerializationContext context)

It seems that in NJsonSchema 11.0.0 the JsonSchemaGeneratorSettings class in particular has had some major (both source and binary) breaking changes which currently affect the code in the Confluent.SchemaRegistry.Serdes.Json library:

• The ActualSerializerSettings and SerializerSettings properties no longer exist.
• The class is now abstract and cannot be instantiated.

.NET 7 will reach end of support on May 14th this year, and besides that we'd very much like to migrate to .NET 8 for various reasons.

What is the recommended approach here? Would it be possible for Confluent.SchemaRegistry.Serdes.Json to migrate to NJsonSchema version 11.0.0? I'd be more than happy to open a PR regarding this issue, but thought it'd be a good idea to discuss here first.

How to reproduce

1. Install both Confluent.SchemaRegistry.Serdes.Json 2.3.0 and NJsonSchema 11.0.0.
2. Produce a JSON message to Kafka.

<Project Sdk="Microsoft.NET.Sdk">

    <PropertyGroup>
        <OutputType>Exe</OutputType>
        <TargetFramework>net8.0</TargetFramework>
        <ImplicitUsings>enable</ImplicitUsings>
        <Nullable>enable</Nullable>
    </PropertyGroup>

    <ItemGroup>
        <PackageReference Include="Confluent.SchemaRegistry.Serdes.Json" Version="2.3.0" />
        <PackageReference Include="NJsonSchema" Version="11.0.0" />
    </ItemGroup>

</Project>

using Confluent.Kafka;
using Confluent.SchemaRegistry;
using Confluent.SchemaRegistry.Serdes;

SchemaRegistryConfig schemaRegistryConfig = new() { Url = "test" };
ISchemaRegistryClient schemaRegistryClient = new CachedSchemaRegistryClient(schemaRegistryConfig);

ProducerConfig producerConfig = new() { BootstrapServers = "localhost:9092" };
IProducer<Null, TestJsonMessage> producer = new ProducerBuilder<Null, TestJsonMessage>(producerConfig)
	.SetValueSerializer(new JsonSerializer<TestJsonMessage>(schemaRegistryClient))
	.Build();

await producer.ProduceAsync("test", new() { Value = new("test") });

public record TestJsonMessage(string Id);

The program crashes with the following exception:

Unhandled exception. Confluent.Kafka.ProduceException`2[Confluent.Kafka.Null,TestJsonMessage]: Local: Value serialization error                                                   
 ---> System.MissingMethodException: Method not found: 'Newtonsoft.Json.JsonSerializerSettings NJsonSchema.Generation.JsonSchemaGeneratorSettings.get_ActualSerializerSettings()'.
   at Confluent.SchemaRegistry.Serdes.JsonSerializer`1.SerializeAsync(T value, SerializationContext context)                                                                      
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)                                                                    
   at Confluent.SchemaRegistry.Serdes.JsonSerializer`1.SerializeAsync(T value, SerializationContext context)                                                                      
   at Confluent.Kafka.Producer`2.ProduceAsync(TopicPartition topicPartition, Message`2 message, CancellationToken cancellationToken)                                              
   --- End of inner exception stack trace ---                                                                                            

Checklist

Please provide the following information:

• A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
• Confluent.Kafka nuget version.
• Apache Kafka version.
• Client configuration.
• Operating system.
• Provide logs (with "debug" : "..." as necessary in configuration).
• Provide broker log excerpts.
• Critical issue.

[KB-RBullock]

We are experiencing this issue when attempting to update to .NET 8 also

[SukharevAndrey]

@NoahStolk Confluent will do nothing. They do not actively support most of their open-source libraries. Just look at elasticsearch and jdbc connectors for instance: they completely ignore issues and PRs.
I suggest you making a copy of NJsonSchema related library code and fix it yourself.

[NoahStolk]

@NoahStolk Confluent will do nothing. They do not actively support most of their open-source libraries. Just look at elasticsearch and jdbc connectors for instance: they completely ignore issues and PRs. I suggest you making a copy of NJsonSchema related library code and fix it yourself.

That's a shame, and I have already done that and upgraded to .NET 8 successfully with minimal changes. I would however prefer to keep this issue open for anyone else who might run into the same problem.

The Confluent.SchemaRegistry.Serdes.Json library is not necessarily complex so it's fairly easy to make a copy and make the necessary changes (especially in our case since we were only using about 20% of the library's code). It's not ideal though.

[anchitj]

Thanks for reporting. I'll check before the new version release if we can upgrade the NJsonSchema. Please open up a PR if you can, I'll take a look.

[NoahStolk]

Thanks @anchitj, I'll set up a branch with my current work, and see if I can port the remaining parts of the library to NJsonSchema 11 later this week. 👍

[NoahStolk]

I have completed my PR. Build and unit tests are succeeding. Please let me know if there's anything else I can do to get this merged. It would be ideal if the change could be included in a 3.0.0 release.

[NoahStolk]

Hi @anchitj, are you able to give us an indication of when we can expect a new release? I imagine the PR would not get merged until a new release is scheduled. Is that correct?

[KB-RBullock]

Anything that could be done to expedite this fix/release would be much appreciated by the community, it is a show stopper for .NET8 migration

Thanks in advance