Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SystemTextJsonSerializer: Adds Default STJ Serializer #4332

Open
wants to merge 18 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
bf78e7c
Code changes to add default STJ serializer.
kundadebdatta Feb 27, 2024
306b929
Code changes to add tests for STJ serializer.
kundadebdatta Feb 28, 2024
49844eb
Code changes to rename files.
kundadebdatta Mar 11, 2024
d49dff9
Code changes to clean up files.
kundadebdatta Mar 11, 2024
2d17302
Code changes to add more tests.
kundadebdatta Mar 11, 2024
c78093a
Merge branch 'master' into users/kundadebdatta/4330_add_default_stj_s…
kundadebdatta Mar 11, 2024
004d08d
Code changes to update respective contracts.
kundadebdatta Mar 11, 2024
3a728b1
Code changes to add more documentation.
kundadebdatta Mar 20, 2024
089095c
Code changes to add LINQ tests. Addressing few review comments.
kundadebdatta Apr 11, 2024
8b208de
Code changes to fix build failures.
kundadebdatta Apr 11, 2024
ca3b6a6
Code changes to update contract files.
kundadebdatta Apr 12, 2024
73178e0
Code changes to add more LINQ specific test cases.
kundadebdatta Apr 15, 2024
683ca13
Code changes to address review comments.
kundadebdatta Apr 15, 2024
61171cd
Merge branch 'master' into users/kundadebdatta/4330_add_default_stj_s…
kundadebdatta Apr 15, 2024
5df9c07
Merge branch 'master' into users/kundadebdatta/4330_add_default_stj_s…
kundadebdatta May 9, 2024
cdbf98c
Code changes to port some of the optimizations from Maya.
kundadebdatta May 9, 2024
4938ac3
Merge branch 'master' into users/kundadebdatta/4330_add_default_stj_s…
kundadebdatta May 10, 2024
37009ca
Code changes to update the remarks since the changes from maya will c…
kundadebdatta May 10, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -1043,7 +1043,7 @@ public async Task VerifyDekOperationWithSystemTextSerializer()
DefaultIgnoreCondition = System.Text.Json.Serialization.JsonIgnoreCondition.WhenWritingNull
};

CosmosSystemTextJsonSerializer cosmosSystemTextJsonSerializer = new CosmosSystemTextJsonSerializer(jsonSerializerOptions);
LegacyEncryptionTests.CosmosSystemTextJsonSerializer cosmosSystemTextJsonSerializer = new (jsonSerializerOptions);

CosmosClient clientWithCosmosSystemTextJsonSerializer = TestCommon.CreateCosmosClient(builder => builder
.WithCustomSerializer(cosmosSystemTextJsonSerializer)
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,130 @@
// ------------------------------------------------------------
// Copyright (c) Microsoft Corporation. All rights reserved.
// ------------------------------------------------------------

namespace Microsoft.Azure.Cosmos
{
using System;
using System.IO;
using System.Reflection;
using System.Text.Json;
using System.Text.Json.Serialization;

/// <summary>
/// This class provides a default implementation of System.Text.Json Cosmos Linq Serializer.
/// </summary>
public class CosmosSystemTextJsonSerializer : CosmosLinqSerializer
kundadebdatta marked this conversation as resolved.
Show resolved Hide resolved
{
/// <summary>
/// A read-only instance of <see cref="JsonSerializerOptions"/>.
/// </summary>
private readonly JsonSerializerOptions jsonSerializerOptions;

/// <summary>
/// Creates an instance of <see cref="CosmosSystemTextJsonSerializer"/>
/// with the default values for the Cosmos SDK
/// </summary>
/// <param name="jsonSerializerOptions">An instance of <see cref="JsonSerializerOptions"/> containing the json serialization options.</param>
public CosmosSystemTextJsonSerializer(
JsonSerializerOptions jsonSerializerOptions)
{
this.jsonSerializerOptions = jsonSerializerOptions;
}

/// <inheritdoc/>
public override T FromStream<T>(Stream stream)
{
if (stream == null)
throw new ArgumentNullException(nameof(stream));

if (stream.CanSeek && stream.Length == 0)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is canSeek is not really required right?

{
return default;
}

if (typeof(Stream).IsAssignableFrom(typeof(T)))
{
return (T)(object)stream;
}

using (stream)
{
using StreamReader reader = new (stream);
return JsonSerializer.Deserialize<T>(reader.ReadToEnd(), this.jsonSerializerOptions);
kundadebdatta marked this conversation as resolved.
Show resolved Hide resolved
}
}

/// <inheritdoc/>
public override Stream ToStream<T>(T input)
{
MemoryStream streamPayload = new ();
using Utf8JsonWriter writer = new (streamPayload);

JsonSerializer.Serialize(
writer: writer,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its great that I don't need to care as this will be an impl detail deep in a library but this formatting/named params is way overkill and makes it look like some really exciting stuff is going on, which there isnt
is the intention to disambiguate from some adjacent overload?
in general I'd be nice if this code was scannable so people can compare their impls quickly before deleting them in favor of this one

value: input,
options: this.jsonSerializerOptions);

streamPayload.Position = 0;
return streamPayload;
}

/// <summary>
/// Convert a MemberInfo to a string for use in LINQ query translation.
/// </summary>
/// <param name="memberInfo">Any MemberInfo used in the query.</param>
/// <returns>A serialized representation of the member.</returns>
/// <remarks>
/// Note that this is just a default implementation which handles the basic scenarios. Any <see cref="JsonSerializerOptions"/> passed in
/// here are not going to be reflected in SerializeMemberName(). For example, if customers passed in a JsonSerializerOption such as below
/// <code language="c#">
/// <![CDATA[
/// JsonSerializerOptions options = new()
/// {
/// PropertyNamingPolicy = JsonNamingPolicy.CamelCase
kundadebdatta marked this conversation as resolved.
Show resolved Hide resolved
/// }
/// ]]>
/// </code>
/// This would not be honored by SerializeMemberName() unless it included special handling for this, for example.
/// <code language="c#">
/// <![CDATA[
/// public override string SerializeMemberName(MemberInfo memberInfo)
/// {
/// JsonExtensionDataAttribute jsonExtensionDataAttribute =
/// memberInfo.GetCustomAttribute<JsonExtensionDataAttribute>(true);
/// if (jsonExtensionDataAttribute != null)
/// {
/// return null;
/// }
/// JsonPropertyNameAttribute jsonPropertyNameAttribute = memberInfo.GetCustomAttribute<JsonPropertyNameAttribute>(true);
/// if (!string.IsNullOrEmpty(jsonPropertyNameAttribute?.Name))
/// {
/// return jsonPropertyNameAttribute.Name;
/// }
/// return System.Text.Json.JsonNamingPolicy.CamelCase.ConvertName(memberInfo.Name);
/// }
/// ]]>
/// </code>
/// To handle such scenarios, please create a custom serializer which inherits from the <see cref="CosmosSystemTextJsonSerializer"/> and overrides the
/// SerializeMemberName to add any special handling.
/// </remarks>
public override string SerializeMemberName(MemberInfo memberInfo)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've updated the STJ sample code and made some improvements to this function in this PR #4420. Could you update this function to look like this - https://github.com/Azure/azure-cosmos-dotnet-v3/blob/master/Microsoft.Azure.Cosmos.Samples/Usage/SystemTextJson/CosmosSystemTextJsonSerializer.cs ? Thanks!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Maya-Painter. We are trying to gather some feedback from the .NET team on this. I will take your changes as well, and update the PR.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The JsonTypeInfo/JsonPropertyInfo types contain authoritative information on the JSON contract used by a particular type/property. See https://learn.microsoft.com/en-us/dotnet/standard/serialization/system-text-json/custom-contracts for more details.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @eiriktsarpalis !

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your suggestion @eiriktsarpalis! Looks like, these (JsonSerializerOptions.TryGetTypeInfo, custom contracts) are applicable for the latest versions of the System.Text.JSON package, and it's not available in 4.6.0.

In order to use the System.Text.Json (STJ) serializer, we currently depend on the Azure.Core nuget package (specifically for this class JsonObjectSerializer). The current version of Azure.Core library, referred in our v3 .NET SDK is 1.19.0, which is quite old. This internally uses System.Text.Json version 4.6.0.

I am planning to keep this suggestion as follow-up task 4479 to upgrade the Azure.Core library to the latest. Even the latest version of Azure.Core 1.39.0 library uses the STJ version 4.7.2.

image

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about only make this type public once is fully addressed?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fact that Azure.Core 'uses' (gets by with) an old version does not of itself mean that this lib can't take a dependency on a higher version. I'd ask, what version would simplify and/or significantly improve perf here? Then validate the answer by lookign at package download counts at https://www.nuget.org/packages/system.text.json.
For instance 6.0.1 has a net6.0 variant and seems to be in popular use, and is only 18 months newer - it's not like STJ is known to be a pain to upgrade.
But of course, I'm just being a devil's advocate; there's lots of silent votes for the status quo!
Hopefully there will be a security reason along to trigger a forced upgrade in due course ;)
The other benefit of taking a higher dep is that perf in general has been on an upward trajectory, and the perceived experience of the SDK would hence also improve for people that don't explicitly pin latest in their apps

{
JsonExtensionDataAttribute jsonExtensionDataAttribute =
memberInfo.GetCustomAttribute<JsonExtensionDataAttribute>(true);

if (jsonExtensionDataAttribute != null)
{
return null;
}

JsonPropertyNameAttribute jsonPropertyNameAttribute = memberInfo.GetCustomAttribute<JsonPropertyNameAttribute>(true);

string memberName = !string.IsNullOrEmpty(jsonPropertyNameAttribute?.Name)
? jsonPropertyNameAttribute.Name
: memberInfo.Name;

return memberName;
}
}
}
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
<Results>
<Result>
<Input>
<Description><![CDATA[Avg, Custom serializer: True]]></Description>
<Description><![CDATA[Avg, Serializer Name: SystemTextJsonLinqSerializer]]></Description>
<Expression><![CDATA[query.Average(doc => doc.NumericField), Object)]]></Expression>
</Input>
<Output>
Expand All @@ -12,7 +12,7 @@ FROM root]]></SqlQuery>
</Result>
<Result>
<Input>
<Description><![CDATA[Sum, Custom serializer: True]]></Description>
<Description><![CDATA[Sum, Serializer Name: SystemTextJsonLinqSerializer]]></Description>
<Expression><![CDATA[query.Sum(doc => doc.NumericField), Object)]]></Expression>
</Input>
<Output>
Expand All @@ -23,7 +23,7 @@ FROM root]]></SqlQuery>
</Result>
<Result>
<Input>
<Description><![CDATA[Select many -> Filter -> Select -> Average, Custom serializer: True]]></Description>
<Description><![CDATA[Select many -> Filter -> Select -> Average, Serializer Name: SystemTextJsonLinqSerializer]]></Description>
<Expression><![CDATA[query.SelectMany(doc => doc.ArrayField.Where(m => ((m % 3) == 0)).Select(m => m)).Average(), Object)]]></Expression>
</Input>
<Output>
Expand All @@ -36,7 +36,7 @@ WHERE ((m0 % 3) = 0)]]></SqlQuery>
</Result>
<Result>
<Input>
<Description><![CDATA[Select number -> Skip -> Count, Custom serializer: True]]></Description>
<Description><![CDATA[Select number -> Skip -> Count, Serializer Name: SystemTextJsonLinqSerializer]]></Description>
<Expression><![CDATA[query.Select(f => f.NumericField).Skip(2).Count(), Object)]]></Expression>
</Input>
<Output>
Expand All @@ -63,7 +63,7 @@ FROM root]]></SqlQuery>
</Result>
<Result>
<Input>
<Description><![CDATA[Avg, Custom serializer: False]]></Description>
<Description><![CDATA[Avg, Serializer Name: SystemTextJsonSerializer]]></Description>
<Expression><![CDATA[query.Average(doc => doc.NumericField), Object)]]></Expression>
</Input>
<Output>
Expand All @@ -74,7 +74,7 @@ FROM root]]></SqlQuery>
</Result>
<Result>
<Input>
<Description><![CDATA[Sum, Custom serializer: False]]></Description>
<Description><![CDATA[Sum, Serializer Name: SystemTextJsonSerializer]]></Description>
<Expression><![CDATA[query.Sum(doc => doc.NumericField), Object)]]></Expression>
</Input>
<Output>
Expand All @@ -85,7 +85,7 @@ FROM root]]></SqlQuery>
</Result>
<Result>
<Input>
<Description><![CDATA[Select many -> Filter -> Select -> Average, Custom serializer: False]]></Description>
<Description><![CDATA[Select many -> Filter -> Select -> Average, Serializer Name: SystemTextJsonSerializer]]></Description>
<Expression><![CDATA[query.SelectMany(doc => doc.ArrayField.Where(m => ((m % 3) == 0)).Select(m => m)).Average(), Object)]]></Expression>
</Input>
<Output>
Expand All @@ -98,7 +98,69 @@ WHERE ((m0 % 3) = 0)]]></SqlQuery>
</Result>
<Result>
<Input>
<Description><![CDATA[Select number -> Skip -> Count, Custom serializer: False]]></Description>
<Description><![CDATA[Select number -> Skip -> Count, Serializer Name: SystemTextJsonSerializer]]></Description>
<Expression><![CDATA[query.Select(f => f.NumericField).Skip(2).Count(), Object)]]></Expression>
</Input>
<Output>
<SqlQuery><![CDATA[
SELECT VALUE COUNT(1)
FROM (
SELECT VALUE root["NumericField"]
FROM root
OFFSET 2 LIMIT 2147483647) AS r0
]]></SqlQuery>
<ErrorMessage><![CDATA[Status Code: BadRequest,{"errors":[{"severity":"Error","location":{"start":72,"end":97},"code":"SC2204","message":"'OFFSET LIMIT' clause is not supported in subqueries."}]},0x800A0B00]]></ErrorMessage>
</Output>
</Result>
<Result>
<Input>
<Description><![CDATA[Select number -> Min w/ mapping]]></Description>
<Expression><![CDATA[query.Select(doc => doc.NumericField).Min(num => num), Object)]]></Expression>
</Input>
<Output>
<SqlQuery><![CDATA[
SELECT VALUE MIN(root["NumericField"])
FROM root]]></SqlQuery>
</Output>
</Result>
<Result>
<Input>
<Description><![CDATA[Avg, Serializer Name: CosmosSystemTextJsonSerializer]]></Description>
<Expression><![CDATA[query.Average(doc => doc.NumericField), Object)]]></Expression>
</Input>
<Output>
<SqlQuery><![CDATA[
SELECT VALUE AVG(root["NumericField"])
FROM root]]></SqlQuery>
</Output>
</Result>
<Result>
<Input>
<Description><![CDATA[Sum, Serializer Name: CosmosSystemTextJsonSerializer]]></Description>
<Expression><![CDATA[query.Sum(doc => doc.NumericField), Object)]]></Expression>
</Input>
<Output>
<SqlQuery><![CDATA[
SELECT VALUE SUM(root["NumericField"])
FROM root]]></SqlQuery>
</Output>
</Result>
<Result>
<Input>
<Description><![CDATA[Select many -> Filter -> Select -> Average, Serializer Name: CosmosSystemTextJsonSerializer]]></Description>
<Expression><![CDATA[query.SelectMany(doc => doc.ArrayField.Where(m => ((m % 3) == 0)).Select(m => m)).Average(), Object)]]></Expression>
</Input>
<Output>
<SqlQuery><![CDATA[
SELECT VALUE AVG(m0)
FROM root
JOIN m0 IN root["ArrayField"]
WHERE ((m0 % 3) = 0)]]></SqlQuery>
</Output>
</Result>
<Result>
<Input>
<Description><![CDATA[Select number -> Skip -> Count, Serializer Name: CosmosSystemTextJsonSerializer]]></Description>
<Expression><![CDATA[query.Select(f => f.NumericField).Skip(2).Count(), Object)]]></Expression>
</Input>
<Output>
Expand Down