Mystic MissingMethodException

[tkirill]

NEST/Elasticsearch.Net version: 7.9

Elasticsearch version: 7.16.3

.NET runtime version: net6, net48

Operating system version: Windows 10.0.17763

Description of the problem including expected versus actual behavior:

It is a very strange problem. I suddenly started to get various MissingMethodException from elasticsearch-net. It looks very similar to #6144. As in #6144 exceptions came out of nowhere and aren't consistent across replicas of my application. I didn't managed to reproduce the problem. I understand that it is hard to investigate without steps to reproduce. But I thought that it is worth to try to provide a detailed information about this mystic problem to you. Maybe you will have some ideas from your experience that will guide my further investigation.

I have two kinds of the error. 1️⃣ System.MissingMethodException: Method not found: 'Nest.ceAggregation Nest.IAggregationContainer.get_GeoDistance()' and 2️⃣ System.MissingMethodException: Method not found: 'Void Elasticsearch.Net.Utf8Json.IJsonFormatter`1.IGeoDistanceAggregation(Elasticsearch.Net.Utf8Json.JsonWriter ByRef, !0, Elasticsearch.Net.Utf8Json.IJsonFormatterResolver)'

Stacktrace for 1️⃣:

Elasticsearch.Net.UnexpectedElasticsearchClientException: Method not found: 'Nest.ceAggregation Nest.IAggregationContainer.get_GeoDistance()'.
    at Elasticsearch.Net.Transport`1+<RequestAsync>d__15`1.MoveNext(C:\Users\russc\source\elasticsearch-net\src\Elasticsearch.Net\Transport\Transport.cs:153:7)
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification()
    at Nest.ElasticClient+<SearchAsync>d__254`1.MoveNext(C:\Users\russc\source\elasticsearch-net\src\Nest\ElasticClient.NoNamespace.cs:1003:31)
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification()
    at Nest.ElasticClient+<SearchAsync>d__252`1.MoveNext(C:\Users\russc\source\elasticsearch-net\src\Nest\ElasticClient.NoNamespace.cs:989:31)
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification()
    at
   <... Application code ...>
  Caused by: System.MissingMethodException: Method not found: 'Nest.ceAggregation Nest.IAggregationContainer.get_GeoDistance()'.
      at Elasticsearch.Net.Nest_IAggregationContainerFormatter3.Serialize()
      at Nest.VerbatimDictionaryKeysBaseFormatter`3.Serialize(C:\Users\russc\source\elasticsearch-net\src\Nest\Aggregations\VerbatimDictionaryKeysFormatter.cs:30:4)
      at Elasticsearch.Net.Nest_ISearchRequestFormatter1.Serialize()
      at Elasticsearch.Net.Utf8Json.JsonSerializer+<SerializeAsync>d__11`1.MoveNext(C:\Users\russc\source\elasticsearch-net\src\Elasticsearch.Net\Utf8Json\JsonSerializer.cs:145:9)
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification()
      at Elasticsearch.Net.SerializableData`1+<WriteAsync>d__4.MoveNext(C:\Users\russc\source\elasticsearch-net\src\Elasticsearch.Net\Transport\SerializableData.cs:47:3)
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification()
      at Elasticsearch.Net.HttpConnection+<SetContentAsync>d__24.MoveNext(C:\Users\russc\source\elasticsearch-net\src\Elasticsearch.Net\Connection\HttpConnection.cs:374:3)
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification()
      at Elasticsearch.Net.HttpConnection+<RequestAsync>d__14`1.MoveNext(C:\Users\russc\source\elasticsearch-net\src\Elasticsearch.Net\Connection\HttpConnection.cs:165:3)
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification()
      at Elasticsearch.Net.RequestPipeline+<CallElasticsearchAsync>d__57`1.MoveNext(C:\Users\russc\source\elasticsearch-net\src\Elasticsearch.Net\Transport\Pipeline\RequestPipeline.cs:200:3)
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification()
      at Elasticsearch.Net.Transport`1+<RequestAsync>d__15`1.MoveNext(C:\Users\russc\source\elasticsearch-net\src\Elasticsearch.Net\Transport\Transport.cs)

Stacktrace for 2️⃣:

Elasticsearch.Net.UnexpectedElasticsearchClientException: Method not found: 'Void Elasticsearch.Net.Utf8Json.IJsonFormatter`1.IGeoDistanceAggregation(Elasticsearch.Net.Utf8Json.JsonWriter ByRef, !0, Elasticsearch.Net.Utf8Json.IJsonFormatterResolver)'.
    at Elasticsearch.Net.Transport`1+<RequestAsync>d__15`1.MoveNext(C:\Users\russc\source\elasticsearch-net\src\Elasticsearch.Net\Transport\Transport.cs:153:7)
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification()
    at Nest.ElasticClient+<SearchAsync>d__254`1.MoveNext(C:\Users\russc\source\elasticsearch-net\src\Nest\ElasticClient.NoNamespace.cs:1003:31)
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification()
    at Nest.ElasticClient+<SearchAsync>d__252`1.MoveNext(C:\Users\russc\source\elasticsearch-net\src\Nest\ElasticClient.NoNamespace.cs:989:31)
    at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
    at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess()
    at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification()
    at
    <... Application code ...>
  Caused by: System.MissingMethodException: Method not found: 'Void Elasticsearch.Net.Utf8Json.IJsonFormatter`1.IGeoDistanceAggregation(Elasticsearch.Net.Utf8Json.JsonWriter ByRef, !0, Elasticsearch.Net.Utf8Json.IJsonFormatterResolver)'.
      at Elasticsearch.Net.Nest_IQueryContainerFormatter2.Serialize()
      at Elasticsearch.Net.Nest_ISearchRequestFormatter1.Serialize()
      at Elasticsearch.Net.Utf8Json.JsonSerializer+<SerializeAsync>d__11`1.MoveNext(C:\Users\russc\source\elasticsearch-net\src\Elasticsearch.Net\Utf8Json\JsonSerializer.cs:145:9)
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification()
      at Elasticsearch.Net.SerializableData`1+<WriteAsync>d__4.MoveNext(C:\Users\russc\source\elasticsearch-net\src\Elasticsearch.Net\Transport\SerializableData.cs:47:3)
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification()
      at Elasticsearch.Net.HttpConnection+<SetContentAsync>d__24.MoveNext(C:\Users\russc\source\elasticsearch-net\src\Elasticsearch.Net\Connection\HttpConnection.cs:374:3)
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification()
      at Elasticsearch.Net.HttpConnection+<RequestAsync>d__14`1.MoveNext(C:\Users\russc\source\elasticsearch-net\src\Elasticsearch.Net\Connection\HttpConnection.cs:165:3)
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification()
      at Elasticsearch.Net.RequestPipeline+<CallElasticsearchAsync>d__57`1.MoveNext(C:\Users\russc\source\elasticsearch-net\src\Elasticsearch.Net\Transport\Pipeline\RequestPipeline.cs:200:3)
      at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
      at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess()
      at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification()
      at Elasticsearch.Net.Transport`1+<RequestAsync>d__15`1.MoveNext(C:\Users\russc\source\elasticsearch-net\src\Elasticsearch.Net\Transport\Transport.cs)

Important observations and facts about problem appearance:

1. I didn't update the client for a few years. It worked perfect (thank you for such great library! 🙂).
2. The problem appears relatively rarely. Sometimes after an another release of my application 1-2 replicas out of many suddenly start to throw MissingMethodException after restart.
3. If replica start to throw MissingMethodException it won't stop. Only restart helps.

Important obsevations and facts that I managed to gather myself:

1. I tried to reproduce the error repeating the request in single thread and in 10 threads for five minutes. The error was not reproduced.
2. I think I found the code which emits the code that throws the exceptions 😵 get_ call from 1️⃣ is emitted here (EmitLoadValue). And call of a method with such arguments from 2️⃣ is emitted here (EmitInfo.Serialize).
3. Both errors are really strange. I can't understand from the code why it happens. 1️⃣ -- IAggregationContainer indeed has getter for a GeoDistance. Why runtime suddenly can't find it? 2️⃣ -- guessing from exception we try to call method IGeoDistanceAggregation on IJsonFormatter<T> which indeed doesn't exist. And IGeoDistanceAggregation is an interface and definitely not a method! :) And we emit a call to IJsonFormatter.Serialize. So why do we see in exception a call to IJsonFormatter<T>.IGeoDistanceAggregation with the same parameters as IJsonFormatter<T>.Serialize has?
4. The code looks deterministic. So it should either always work or always fail. I can't understand what introduces instability. Maybe a race condition during initialisation of the client? But the problem was not reproduced in multithreaded test. Maybe it is different version of the Elasticsearch.net.dll? But the same set of binaries works good after the restart on the same host. Maybe the client isn't compatible with Elasticsearch version? But the error appears during request serialisation.

Steps to reproduce: ➖

[asal-hesehus]

I have also seen this issue. It happen rarely but the application needs to be restarted.