Cannot run knn query using elasticClient.RequestResponseSerializer

[giannik]

Im trying to run a knn query in version 8.9 of dotnet client by passing the following json string into elastic client serializer as below :

json query string

{
  "knn": {    
    "field": "ContentItem_Vector",
    "k": 10,
    "num_candidates": 100,
    "query_vector": [0.1212, 0.6565 ......]
  }
}

passing json query into stream and then _elasticClient?.RequestResponseSerializer
error on _elasticClient?.RequestResponseSerialize

  try
                    {
                        using var stream = new MemoryStream(Encoding.UTF8.GetBytes(query));
                        var deserializedSearchRequest = _elasticClient?.RequestResponseSerializer.Deserialize<SearchRequest>(stream);

                        var searchRequest = new SearchRequest(_indexPrefix + indexName)
                        {
                            Query = deserializedSearchRequest.Query,
                            From = deserializedSearchRequest.From,
                            Size = deserializedSearchRequest.Size,
                            Fields = deserializedSearchRequest.Fields,
                            Sort = deserializedSearchRequest.Sort,
                            Source = deserializedSearchRequest.Source,
                        };

                        var searchResponse = await _elasticClient.SearchAsync<Dictionary<string, object>>(searchRequest);
                        var hits = new List<Dictionary<string, object>>();

                        foreach (var hit in searchResponse.Hits)
                        {
                            if (hit.Fields != null)
                            {
                                var row = new Dictionary<string, object>();

                                foreach (var keyValuePair in hit.Fields)
                                {
                                    row[keyValuePair.Key] = keyValuePair.Value.ToString();
                                }

                                hits.Add(row);
                            }
                        }

                        if (searchResponse.IsValidResponse)
                        {
                            elasticTopDocs.Count = searchResponse.Total;
                            elasticTopDocs.TopDocs = [.. searchResponse.Documents];
                            elasticTopDocs.Fields = hits;
                        }
                        else
                        {
                            _logger.LogError("Received failure response from Elasticsearch: {ServerError}", searchResponse.ElasticsearchServerError?.ToString());
                        }
                    }
                    catch (Exception ex)
                    {
                        _logger.LogError(ex, "Error while querying elastic with exception: {Message}", ex.Message);
                    }

error :

System.Text.Json.JsonException: The JSON value could not be converted to System.Collections.Generic.ICollection`1[Elastic.Clients.Elasticsearch.KnnQuery]. Path: $ | LineNumber: 0 | BytePositionInLine: 1.
   at System.Text.Json.ThrowHelper.ThrowJsonException_DeserializeUnableToConvertValue(Type propertyType)
   at System.Text.Json.Serialization.JsonCollectionConverter`2.OnTryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, TCollection& value)
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value, Boolean& isPopulatedValue)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.JsonSerializer.Read[TValue](Utf8JsonReader& reader, JsonTypeInfo`1 jsonTypeInfo)
   at Elastic.Clients.Elasticsearch.SearchRequestConverter.Read(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options) in /_/src/Elastic.Clients.Elasticsearch/_Generated/Api/SearchRequest.g.cs:line 248
   at System.Text.Json.Serialization.JsonConverter`1.TryRead(Utf8JsonReader& reader, Type typeToConvert, JsonSerializerOptions options, ReadStack& state, T& value, Boolean& isPopulatedValue)
   at System.Text.Json.Serialization.JsonConverter`1.ReadCore(Utf8JsonReader& reader, JsonSerializerOptions options, ReadStack& state)
   at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.ContinueDeserialize(ReadBufferState& bufferState, JsonReaderState& jsonReaderState, ReadStack& readStack)
   at System.Text.Json.Serialization.Metadata.JsonTypeInfo`1.Deserialize(Stream utf8Json)
   at Elastic.Clients.Elasticsearch.Serialization.SystemTextJsonSerializer.Deserialize[T](Stream stream) in /_/src/Elastic.Clients.Elasticsearch.Shared/Serialization/SystemTextJsonSerializer.cs:line 71
   at OrchardCore.Search.Elasticsearch.Core.Services.ElasticQueryService.SearchAsync(String indexName, String query) in C:\IntelliDev1\src\OrchardCore\OrchardCore.Search.Elasticsearch.Core\Services\ElasticQueryService.cs:line 59

It seems that it does not have the serializers for the new query type.
It works for other elastic query types like multi match but not this new knn type.

I wish to keep using this generic way and not change my code based on query type.

if i run it using the static types it works

--this works
     var searchResponse = await _elasticClient.SearchAsync<Dictionary<string, object>>(s =>
                                s.Index(_indexPrefix + indexName)
                                    .Knn(qd =>
                                    {
                                        qd.k(10)
                                        .NumCandidates(100)
                                            .Field("ContentItem_Vector")
                                            .QueryVector(queryVector);
                                    }),
                            default)
                        .ConfigureAwait(false);

[flobernd]

Hi @giannik, You are right, KNN queries are not completeley implemented yet. I can not give you an exact ETA for that, as it currently does not have very high priority.

Besides that, please note, that deserialization of requests is not officially supported (and might not work with some requests). The client usually only serializes requests and deserializes responses. Roundtripping is at best partially implemented 🙂

[giannik]

@flobernd thanks for the notice.
do you have a recomended generic approach to run queries in the dotnet client based on the json query body given as a string ?
for example easily identify what type of query it is (that derives from base class searchQuery ) from the json string and generically be able to execute the query in dotnet ?

[flobernd]

@giannik This scenario is not directly supported by the client. If you have query payloads in JSON format, I would probably use the low level transport client to send raw requests (you could probably still deserialize the response to the corresponding CLR type).

There is a chance that the KNN query deserialization might work after some code generation improvements I'm currently working on (earliest release in the end of Q1), but I can't promise anything.

[giannik]

thank you @flobernd .
Can you share any links (docs , examples ) on how to use the low level client in latest dotnet client ?
Yiannis

[flobernd]

@giannik There is a very short example in this migration guide document: https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/migration-guide.html#_workarounds_for_missing_features

Deserializing back to the CLR response types is more complex and I currently don't have any read-to-use documentation for that.

[flobernd]

KNN queries are now fully supported in 8.13.x.

[giannik]

@flobernd thank you very much.