Exception when deserializing quoted numbers with constructors using synchronous API

[nathanpovo]

Description

The following error is getting thrown when I attempt to deserialize a JSON string with quoted numbers (though I specified that numbers can be read from strings in the json serializer options) into a class with a constructor:

System.Text.Json.JsonException
  HResult=0x80131500
  Message=The JSON value could not be converted to System.Nullable`1[System.Int64]. Path: $.album.userplaycount | LineNumber: 8 | BytePositionInLine: 3.
  Source=System.Text.Json
  StackTrace:
   at System.Text.Json.ThrowHelper.ReThrowWithPath(ReadStack& state, Utf8JsonReader& reader, Exception ex)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.JsonSerializer.ReadCore[TValue](JsonConverter jsonConverter, Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.JsonSerializer.ReadCore[TValue](Utf8JsonReader& reader, Type returnType, JsonSerializerOptions options)
   at System.Text.Json.JsonSerializer.Deserialize[TValue](String json, Type returnType, JsonSerializerOptions options)
   at System.Text.Json.JsonSerializer.Deserialize[TValue](String json, JsonSerializerOptions options)
   at GitHubIssue.SynchronousTests.ResponseWithAllPropertiesTest() in C:\Users\nate9\source\repos\LastFM.API\GitHubIssue\SynchronousTests.cs:line 16

  This exception was originally thrown at this call stack:
    [External Code]

Inner Exception 1:
InvalidOperationException: Cannot get the value of a token type 'String' as a number.

Given the following JSON response from an API:

{
  "album": {
    "name": "the name of the album",
    "artist": "the name of the artist",
    "userplaycount": "123",
    "wiki": {
      "summary": "a summary of the album"
    }
  }
}

Where the "album.userplaycount" and the "album.wiki" properties can be omitted in the response.

I have created the following classes:

# nullable enable

public class Result
{
    public Album? Album { get; init; }
}

public class Album
{
    public Album(string name, string artist)
    {
        Name = name;
        Artist = artist;
    }

    public string Name { get; init; }
    public string Artist { get; init; }

    public long? UserPlayCount { get; init; }
}

Note that no property has been created for the "album.wiki" deliberately.

Using the code:

JsonSerializerOptions jsonSerializerOptions = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.AllowReadingFromString,
    PropertyNameCaseInsensitive = true
};

Result result = JsonSerializer.Deserialize<Result>(text, jsonSerializerOptions);

no errors are thrown when the following JSON is deserialized

{
  "album": {
    "name": "Favourite Worst Nightmare",
    "artist": "Arctic Monkeys",
    "userplaycount": "1234"
  }
}

However, if I deserialize the JSON:

{
  "album": {
    "name": "Back in Black",
    "artist": "AC/DC",
    "userplaycount": "123",
    "wiki": {
      "summary": "Back in Black is an album ..."
    }
  }
}

I get the error mentioned above.

No Errors are thrown if:

• The JSON is modified such that the wiki property is moved before the userplaycount property:

{
  "album": {
    "name": "Back in Black",
    "artist": "AC/DC",
    "wiki": {
      "summary": "Back in Black is an album ..."
    },
    "userplaycount": "123"
  }
}

• If the "bad" JSON is deserialized asynchronously with the following code:

JsonSerializerOptions jsonSerializerOptions = new JsonSerializerOptions
{
    NumberHandling = JsonNumberHandling.AllowReadingFromString,
    PropertyNameCaseInsensitive = true
};

using (FileStream SourceStream = File.Open(@"JsonFileDirectory\bad.json", FileMode.Open))
{
    Result result = await JsonSerializer.DeserializeAsync<Result>(SourceStream, jsonSerializerOptions);
}

• If the constructor of the Album class is removed.
• If the userplaycount property is omitted in the API response.

Configuration

I am running this code on:

• .NET 5.0.100-rc.2.20479.15
• Windows 10

Other information

This issue is related to #30255.

I have created a small repo to demonstrate this issue here: https://github.com/nathanpovo/Issue-With-Quoted-Numbers

Workaround

No errors are thrown if the following converter is used with the UserPlayCount property in the album class.

public class JsonConverterNullableInt64 : JsonConverter<long?>
{
    public override long? Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
    {
        if (reader.TokenType == JsonTokenType.String)
        {
            ReadOnlySpan<byte> span = reader.HasValueSequence ? reader.ValueSequence.ToArray() : reader.ValueSpan;

            if (Utf8Parser.TryParse(span, out long number, out int bytesConsumed) && span.Length == bytesConsumed)
            {
                return number;
            }

            if (long.TryParse(reader.GetString(), out number))
            {
                return number;
            }
        }

        return null;
    }

    public override void Write(Utf8JsonWriter writer, long? value, JsonSerializerOptions options)
    {
        throw new NotImplementedException();
    }
}

[layomia]

I was able to repro this and I'm preparing a fix.

[layomia]

The issue here is that the mechanism to honor the number handling of properties during synchronous deserialization, after object construction with parameters isn't properly set. This leads to a situation where custom number handling may not be honored, depending on the ordering of the JSON properties. This failure characteristic is hard to explain/predict and we should consider porting the fix in a servicing release.

The workaround here for 5.0 is to set a custom converter for any failing properties as done in the description above.

[layomia]

Reopening for servicing consideration in 5.0.

[wizofaus]

I'm also having trouble getting System.Text.Json deserialize "true" and "false" into a bool field of an anonymous type, even using JsonNumberHandling.AllowReadingFromString (accepting "bool" is debatably a number). But I rely on this behavior that the Newstonsoft library implements due to the some of the 3rd party APIs we communicate with (that send back bools as "true" or "false", but only accept them as true or false!), and I can't see how to get it to work with System.Text.Json.

[layomia]

Closing as fixed for 5.0 by #45452.

[layomia]

@wizofaus -bool is not considered a .NET "number" type (i.e. an integral numeric type). The JsonNumberHandling feature does not treat it as such.

To support reading quoted booleans with System.Text.Json's serializer, you'll need to register a custom converter which has logic to parse them - https://dotnetfiddle.net/XBS4WZ.

using System;
using System.Text.Json;
using System.Text.Json.Serialization;

public class Program
{
    public static void Main()
    {
        JsonSerializerOptions options = new() { Converters = { new BooleanConverter() } };

        // true/false literals 
        string json = "true";
        bool val = JsonSerializer.Deserialize<bool>(json, options);
        Console.WriteLine(val);

        json = "false";
        val = JsonSerializer.Deserialize<bool>(json, options);
        Console.WriteLine(val);

        // quoted true/false literals
        json = @"""true""";
        val = JsonSerializer.Deserialize<bool>(json, options);
        Console.WriteLine(val);

        json = @"""false""";
        val = JsonSerializer.Deserialize<bool>(json, options);
        Console.WriteLine(val);

        // Invalid data.
        json = @"""True""";
        //json = "1";
        val = JsonSerializer.Deserialize<bool>(json, options);
        Console.WriteLine(val);
    }

    public class BooleanConverter : JsonConverter<bool>
    {
        public override bool Read(ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
        {
            switch (reader.TokenType)
            {
                case JsonTokenType.True:
                    return true;
                case JsonTokenType.False:
                    return false;
                case JsonTokenType.String:
                    return reader.GetString() switch
                    {
                        "true" => true,
                        "false" => false,
                        _ => throw new JsonException()
                    };
                default:
                    throw new JsonException();
            }
        }

        public override void Write(Utf8JsonWriter writer, bool value, JsonSerializerOptions options)
        {
            writer.WriteBooleanValue(value);
        }
    }
}

[wizofaus]

Ah great, thank you @layomia , I figured it might need a custom converter but was only starting to get familiar with the library so assumed it might have built-in functionality.