Client does not support calling App Engine ElastiSearch _search api

[DR9885]

Is your feature request related to a problem? Please describe.
C# client does not support calling app engine elastic search api
ref: https://www.elastic.co/guide/en/app-search/8.12/elasticsearch-search-api-reference.html

Describe the solution you'd like
To be able to pass the app engine to search against:
SearchAsync(x => x.Index("index").Engine("engine").Query())

Describe alternatives you've considered
Option 1 =>
Action: tried to pass url as the index ex: "api/as/v1/engines/example-engine/elasticsearch"
Result: this failed because was url encoded, so the path was incorrect.
caused by this: https://github.com/elastic/elasticsearch-net/blob/7.17/src/Nest/CommonAbstractions/Request/UrlLookup.cs#L59

Option 2 =>
Action: Add engine url route as hostname "https://fakehost-dev.ent.westus2.azure.elastic-cloud.com/api/as/v1/engines/example-engine/elasticsearch"
Result: This didn't work because it just got cut off.

Option 3 => Fork Repo, and add a way to do this.

[flobernd]

Hi @DR9885,

this repository contains the Elasticsearch client.

Enterprise Search and its App Search component is built on top of Elasticsearch, but is a completely dedicated product.

The App Search API is not fully compatible with the Elasticsearch API which means that it will never be supported by this client.

Creating a separate Enterprise Search client is on the roadmap for a while, but so far no progress has been made.

[flobernd]

Could you please show some code for option 2?

For this particular endpoint I don't see a reason why it would not work with the Elasticsearch client. The request and response is mostly just redirected to/from the underlying Elasticsearch instance.

You should be able to pass the full URI - including a path - to the ElasticsearchClientSettings or to the constructor of ElasticsearchClient without it being truncated.

[DR9885]

@flobernd
Client Version 7.17.5

Example Code

const string url = "https://fakehost.ent.westus2.azure.elastic-cloud.com/api/as/v1/engines/fake-engine/elasticsearch";
const string apiKey = "fake-key";

var connectionPool = new StaticConnectionPool(new [] { new Uri(url)});
var settings = new ConnectionSettings(connectionPool)
    .ApiKeyAuthentication(new ApiKeyAuthenticationCredentials(apiKey));
var elasticClient = new ElasticClient(settings);

var searchResponse = elasticClient.Search<SearchOfferView>(s =>
    s.Index("")
        .TypedKeys(null)
        .Query(q => q.MatchAll()));

Error

Unhandled exception. Elasticsearch.Net.UnsupportedProductException: The client noticed that the server is not Elasticsearch and we do not support this unknown product.
   at Elasticsearch.Net.RequestPipeline.FirstPoolUsage(SemaphoreSlim semaphore)
   at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
   at Elasticsearch.Net.ElasticLowLevelClient.DoRequest[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
   at Nest.ElasticClient.DoRequest[TRequest,TResponse](TRequest p, IRequestParameters parameters, Action`2 forceConfiguration)
   at Nest.ElasticClient.Search[TDocument](ISearchRequest request)
   at Nest.ElasticClient.Search[TDocument](Func`2 selector)
   at Program.<Main>$(String[] args) in C:\Users\dr988\Documents\GitHub\TwoEightyCap.Service.Search\benchmark\TwoEightyCap.Service.Search.Benchmark\Program.cs:line 21
   at Program.<Main>(String[] args)

[DR9885]

@flobernd

Was able to override the check.

var connectionPool = new StaticConnectionPool(new [] { new Uri(url)});
var prop = typeof(IConnectionPool).GetProperty("ProductCheckStatus");
prop.SetValue(connectionPool, ProductCheckStatus.ValidProduct, null);

But have new error if I do that

Invalid NEST response built from a unsuccessful (400) low level call on POST: /api/as/v1/engines/fake-engine-search-offer/elasticsearch/_search
# Audit trail of this API call:
 - [1] PingSuccess: Node: https://fakehost-dev.ent.westus2.azure.elastic-cloud.com/api/as/v1/engines/fake-engine-search-offer/ Took: 00:00:01.7453793
 - [2] BadResponse: Node: https://fakehost-dev.ent.westus2.azure.elastic-cloud.com/api/as/v1/engines/fake-engine-search-offer/ Took: 00:00:00.3943435
# OriginalException: Elasticsearch.Net.ElasticsearchClientException: Request failed to execute. Call: Status code 400 from: POST /api/as/v1/engines/fake-engine-search-offer/elasticsearch/_search
# Request:
{"query":{"match_all":{}}}
# Response:

It looks like these are request details being sent

Method: HEAD, RequestUri: 'https://fakehost-dev.ent.westus2.azure.elastic-cloud.com/api/as/v1/engines/fake-engine-search-offer/elasticsearch/', Version: 1.1, Content: <null>, Headers:
{
  Accept: application/json
  User-Agent: elasticsearch-net/7.17.5+34f1b0b42ae4e0227fb1ad9ebeb7ae3406a998ec
  User-Agent: (Microsoft Windows 10.0.22631; .NET 8.0.0; Nest)
  x-elastic-client-meta: es=7.17.5,net=8.0.0,t=7.17.5,a=0,so=8.0.0
  Authorization: ApiKey fake-key
}

Manual curl does work

curl --request "POST" \
--url "https://fakehost-dev.ent.westus2.azure.elastic-cloud.com/api/as/v1/engines/fake-engine-search-offer/elasticsearch/_search" \
--header "Content-Type: application/json" \
--header "Authorization: Bearer fake-key" \
--data '{"query": { "match_all": {} }}'

C# HttpClient Also Works

var client = new HttpClient(new HttpClientHandler(), false);
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", "fake-key");
using StringContent jsonContent = new(
    JsonSerializer.Serialize(new
    {
        query = new { match_all = new {} }
    }),
    Encoding.UTF8,
    "application/json");
var res = await client.PostAsync(new Uri("https://fakehost-dev.ent.westus2.azure.elastic-cloud.com/api/as/v1/engines/fake-engine-search-offer/elasticsearch/_search"), jsonContent, CancellationToken.None);

[DR9885]

@flobernd If I override the Authorization header it works, how can I change the header with nest client?

Works: Authorization: Bearer fake-key
Fails: Authorization: ApiKey fake-key

or can we get the App Engine to accept ApiKey as a type?

[DR9885]

@flobernd
Was able to override the transport authorization, but this is messy

var transportProperty = typeof(ElasticLowLevelClient).GetProperty("Transport", BindingFlags.Instance | BindingFlags.NonPublic);
var transport = (Transport<IConnectionSettingsValues>)transportProperty.GetValue(elasticClient.LowLevel);
transport.Settings.Headers.Add("Authorization", "Bearer fake-key");

[DR9885]

Full working hack

const string url = "https://fake-dev.ent.westus2.azure.elastic-cloud.com/api/as/v1/engines/tec-engine-search-offer/elasticsearch/";
const string apiKey = "fake-key";

var connectionPool = new StaticConnectionPool(new [] { new Uri(url)});
var settings = new ConnectionSettings(connectionPool)
    .ApiKeyAuthentication(new ApiKeyAuthenticationCredentials(apiKey));
var elasticClient = new ElasticClient(settings);

// Hack: Validation
var prop = typeof(IConnectionPool).GetProperty("ProductCheckStatus");
prop.SetValue(connectionPool, ProductCheckStatus.ValidProduct, null);

// Hack: Fix Auth
var transportProperty = typeof(ElasticLowLevelClient).GetProperty("Transport", BindingFlags.Instance | BindingFlags.NonPublic);
var transport = (Transport<IConnectionSettingsValues>)transportProperty.GetValue(elasticClient.LowLevel);
transport.Settings.Headers.Add("Authorization", $"Bearer {apiKey}");

var searchResponse = elasticClient.Search<SearchOfferView>(s =>
    s.Index("")
        .TypedKeys(null)
        .Query(q => q.MatchAll()));

[flobernd]

Thanks for providing a working hack! A similar workaround is probably possible with the 8.* client as well.

For now, I'm going to close this issue as it's related to the Enterprise Search / AppSearch product and not Elasticsearch 🙂