Unable to serialize double value

[h-h-]

Elastic.Clients.Elasticsearch version: 8.1.1

Elasticsearch version: 8.7.1

.NET runtime version: .NET FW 4.7.2

Operating system version: Win10 22H2

Description of the problem including expected versus actual behavior:
Cannot index an instance of a class containing a public property of type double.
Elastic.Transport.UnexpectedTransportException: Unable to serialize double value.

Steps to reproduce:

1. Execute following code/unit test:

using System;
using System.Diagnostics;
using Elastic.Clients.Elasticsearch;
using Elastic.Transport;
using Xunit;

namespace SearchESSequenceResearch.V1
{
	public class MyDecimalClass
	{
		public decimal MyDecimal { get; set; }
	}

	public class MyDoubleClass
	{
		public double MyDouble { get; set; }
	}

	public class TestIndexDouble
	{
		private const string TestMachineUriString = "https://docker.mydomain.com:9280"; // ES 8 in docker
		private readonly Uri _testMachineUri = new Uri(TestMachineUriString);

		[Fact]
		public void TestIndexingDouble()
		{
			ElasticsearchClient client = GetClient();
			var decimalInstance = new MyDecimalClass
			{
				MyDecimal = 123.45m
			};
			try
			{
				client.Index(decimalInstance, "mytestindex");
			}
			catch (Exception e)
			{
				Debug.WriteLine(e);
				throw;
			}
			
			var doubleInstance = new MyDoubleClass
			{
				MyDouble = 123.45d
			};
			try
			{
				client.Index(doubleInstance, "mytestindex");
			}
			catch (Exception e)
			{
				Debug.WriteLine(e);
				throw;
			}
			
		}

		private ElasticsearchClient GetClient()
		{
			var settings = new ElasticsearchClientSettings(_testMachineUri)
				.Authentication(new BasicAuthentication("elastic", "passwordhere"))
				.ServerCertificateValidationCallback((o, cert, chain, errors) => true);
			settings.DisableDirectStreaming();
			var client = new ElasticsearchClient(settings);
			return client;
		}
	}
}```

**Expected behavior**
The item should be serialized and addet to the index.

**Provide `DebugInformation` (if relevant)**:
Elastic.Transport.UnexpectedTransportException: Unable to serialize double value.

Elastic.Transport.UnexpectedTransportException
Unable to serialize double value.
   bei Elastic.Transport.DefaultHttpTransport`1.ThrowUnexpectedTransportException[TResponse](Exception killerException, List`1 seenExceptions, RequestData requestData, TResponse response, RequestPipeline pipeline) in /home/runner/work/elastic-transport-net/elastic-transport-net/src/Elastic.Transport/DefaultHttpTransport.cs:Zeile 328.
   bei Elastic.Transport.DefaultHttpTransport`1.Request[TResponse](HttpMethod method, String path, PostData data, RequestParameters requestParameters) in /home/runner/work/elastic-transport-net/elastic-transport-net/src/Elastic.Transport/DefaultHttpTransport.cs:Zeile 176.
   bei Elastic.Clients.Elasticsearch.ElasticsearchClient.DoRequest[TRequest,TResponse,TRequestParameters](TRequest request, Action`1 forceConfiguration) in /_/src/Elastic.Clients.Elasticsearch/Client/ElasticsearchClient.cs:Zeile 217.

[h-h-]

Same happens for float.
No problems with decimal, int, long, DateTime.

[karl-sjogren]

I submitted a PR for this last week, #7753. It is a bug that only happens when running .NET Framework, not .NET Core or later.

As a temporary workaround you can remove DoubleWithFractionalPortionConverter and FloatWithFractionalPortionConverter from the serializer converters list when constructing the client.

var elasticClientSettings = new ElasticsearchClientSettings(
    new SingleNodePool(new Uri("http://localhost:9200")),
    (defaultSerializer, settings) =>
    new DefaultSourceSerializer(
        settings,
        options => options.Converters.Remove(options.Converters.First(converter => converter.GetType().Name == "DoubleWithFractionalPortionConverter")))) // Need to match on type name since the actual type is internal

[h-h-]

@karl-sjogren Your workaround helped. Thanks a lot! Can continue moving forward with client 8.x

[stevejgordon]

Thanks @karl-sjogren. Will get that PR merged for the next patch release.

[h-h-]

The Exception Unable to serialize double value has gone. But there is a new one:

Elastic.Transport.UnexpectedTransportException: The object or value could not be serialized. Path: $.

Elastic.Transport.UnexpectedTransportException
The object or value could not be serialized. Path: $.
   bei Elastic.Transport.DefaultHttpTransport`1.ThrowUnexpectedTransportException[TResponse](Exception killerException, List`1 seenExceptions, RequestData requestData, TResponse response, RequestPipeline pipeline) in /home/runner/work/elastic-transport-net/elastic-transport-net/src/Elastic.Transport/DefaultHttpTransport.cs:Zeile 327.
   bei Elastic.Transport.DefaultHttpTransport`1.Request[TResponse](HttpMethod method, String path, PostData data, RequestParameters requestParameters) in /home/runner/work/elastic-transport-net/elastic-transport-net/src/Elastic.Transport/DefaultHttpTransport.cs:Zeile 175.
   bei Elastic.Clients.Elasticsearch.ElasticsearchClient.DoRequest[TRequest,TResponse,TRequestParameters](TRequest request, Action`1 forceConfiguration) in /_/src/Elastic.Clients.Elasticsearch/Client/ElasticsearchClient.cs:Zeile 217.
   bei SearchESSequenceResearch.V1.TestIndexDouble.TestIndexingDouble() in C:\GIT\HM\hm3\MS_SEARCH\SearchES\SearchESSequenceResearch\V1\TestIndexDouble.cs:Zeile 56.

System.Text.Json.JsonException
The object or value could not be serialized. Path: $.
   bei System.Text.Json.ThrowHelper.ReThrowWithPath(WriteStack& state, Exception ex)
   bei System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
   bei System.Text.Json.JsonSerializer.WriteCore[TValue](Utf8JsonWriter writer, TValue& value, JsonTypeInfo`1 jsonTypeInfo)
   bei System.Text.Json.JsonSerializer.Serialize[TValue](Utf8JsonWriter writer, TValue value, JsonSerializerOptions options)
   bei Elastic.Clients.Elasticsearch.Serialization.SourceSerialization.Serialize[T](T toSerialize, Utf8JsonWriter writer, Serializer sourceSerializer) in /_/src/Elastic.Clients.Elasticsearch/Serialization/SourceSerialization.cs:Zeile 71.
   bei Elastic.Clients.Elasticsearch.Serialization.CustomJsonWriterConverter`1.Write(Utf8JsonWriter writer, TDocument value, JsonSerializerOptions options) in /_/src/Elastic.Clients.Elasticsearch/Serialization/CustomJsonWriterConverter.cs:Zeile 23.
   bei System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
   bei System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
   bei System.Text.Json.Serialization.JsonConverter`1.WriteCoreAsObject(Utf8JsonWriter writer, Object value, JsonSerializerOptions options, WriteStack& state)
   bei System.Text.Json.JsonSerializer.WriteCoreAsObject(Utf8JsonWriter writer, Object value, JsonTypeInfo jsonTypeInfo)
   bei Elastic.Clients.Elasticsearch.Serialization.SystemTextJsonSerializer.Serialize[T](T data, Stream writableStream, SerializationFormatting formatting) in /_/src/Elastic.Clients.Elasticsearch/Serialization/SystemTextJsonSerializer.cs:Zeile 101.
   bei Elastic.Transport.PostData.SerializableData`1.Write(Stream writableStream, ITransportConfiguration settings) in /home/runner/work/elastic-transport-net/elastic-transport-net/src/Elastic.Transport/Requests/Body/PostData.Serializable.cs:Zeile 42.
   bei Elastic.Transport.HttpWebRequestTransportClient.Request[TResponse](RequestData requestData) in /home/runner/work/elastic-transport-net/elastic-transport-net/src/Elastic.Transport/Components/TransportClient/HttpWebRequestTransportClient.cs:Zeile 52.
   bei Elastic.Transport.DefaultRequestPipeline`1.CallProductEndpoint[TResponse](RequestData requestData) in /home/runner/work/elastic-transport-net/elastic-transport-net/src/Elastic.Transport/Components/Pipeline/DefaultRequestPipeline.cs:Zeile 219.
   bei Elastic.Transport.DefaultHttpTransport`1.Request[TResponse](HttpMethod method, String path, PostData data, RequestParameters requestParameters) in /home/runner/work/elastic-transport-net/elastic-transport-net/src/Elastic.Transport/DefaultHttpTransport.cs:Zeile 163.

System.InvalidOperationException
'}' is invalid following a property name.
   bei System.Text.Json.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource, Int32 currentDepth, Int32 maxDepth, Byte token, JsonTokenType tokenType)
   bei System.Text.Json.Utf8JsonWriter.ValidateEnd(Byte token)
   bei System.Text.Json.Utf8JsonWriter.WriteEndSlow(Byte token)
   bei System.Text.Json.Utf8JsonWriter.WriteEnd(Byte token)
   bei System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state)
   bei System.Text.Json.Serialization.JsonConverter`1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)
   bei System.Text.Json.Serialization.JsonConverter`1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state)

Tested with Client 8.1.3 and Transport 0.4.14

@stevejgordon Could you re-open this issue?

Is there a unit test in the project, so I can see what I'm doing different?

[karl-sjogren]

There are no unit tests for .NET Framework which is why this went unnoticed from the start.

[karl-sjogren]

Yeah looking at this another minute shows that the .NET Framework code is completely broken (or rather, incomplete). Since there were not tests to verify it I missunderstood how severe this was. It didn't just fall through to always throw an exception, the .NET Framework code copies the value into a Span<byte>, but it never writes it to the Utf8JsonWriter.

I can have a look at fixing this in the coming days and when #7769 is done we can add proper tests for this as well.

[flobernd]

#7769 is nearly complete, but I don't want to merge it before the existing tests are passing.

[karl-sjogren]

Of course, I'll prepare a PR for this issue and wait until #7769 is done to make sure that this is fixed for good.