Possibility for Spring Data ElasticSearch to support Spring RestTemplate and Spring Webflux Webclient

[tongdaoqa]

Expected Behavior

In a Spring project, there is a very high likelyhood a Spring RestTemplate, Spring WebFlux Webclient is already on the classpath.

it will be a great improvement for this project, instead of using the default Elasticsearch Java http client, this project can use Spring RestTemplate, Spring WebFlux Webclient that is on a Spring project.

This PR opens this possibility for this project: https://github.com/elastic/elasticsearch-java/pull/584/files

Just to avoid any misunderstanding, this is not an ask to support each and every http client possible and imaginable. Both Spring RestTemplate and Spring Webflux Webclient are very common, and usually present in a default Spring, SpringBoot project. It will be great to be able to leverage those existing clients, now ElasticSearch Java allows it.

Just to avoid misunderstanding, this is not an ask to bring back to life the already deprecated High Level Rest Client https://docs.spring.io/spring-data/elasticsearch/docs/current/reference/html/#elasticsearch.clients.resthighlevelclient or the already deprecated Reactive Client https://docs.spring.io/spring-data/elasticsearch/docs/current/reference/html/#elasticsearch.clients.reactive It is to use the ElasticSearch Java, but with Spring RestTemplate, Spring Webflux Webclient, now this PR offers the possibility to do so

https://github.com/elastic/elasticsearch-java/pull/584/files

Current Behavior

Currently, Spring Data Elasticsearch uses Elasticsearch Java, which uses apache http client.

A great improvement has been made by Elasticsearch Java to allow the support of other http clients.

https://github.com/elastic/elasticsearch-java/pull/584/files

This is an equally great opportunity for this project to be able to wire the usual Spring RestTemplate / Spring WebFlux Webclient instead.

While one can do this replacement on his own, rewriting the code on the SpringBoot/Spring data elasticsearch project, it will be a great improvement if this project can help with the boiler plate code and the configuration, so the application can just configure and ready the use of the http client

Context

How has this issue affected you?

It is usual for a SpringBoot, Spring project to have a preconfigured, tested, tuned Spring RestTemplate Spring Webflux Webclient. However, this project, after the deprecations, does not allow the use of them. This recent PR from Elasticsearch Java is a game changer for Spring based elasticsearch applications.

What are you trying to accomplish?

In most of SpringBoot projects, there is a Webclient, a RestTemplate , which has preconfigured logging, monitoring, circuit breakers, etc... (all part of Spring portfolio) Being able to use those clients within Spring data elasticsearch would be a great plus. Thank you

[sothawo]

Thanks for notifying about this change, I'll have a look of how this can be integrated or extended into Spring Data Elasticsearch.

In a Spring project, there is a very high likelyhood a Spring RestTemplate, Spring WebFlux Webclient is already on the classpath

It's not about the classes being on the classpath, but where the instance of these are created , and Spring itself does not create anything of this, but Spring Boot does. Spring Boot creates such beans if no other library, part of the application has not created one during the startup.

For the combination of Spring Boot and Spring Data Elasticsearch this means, that Spring Boot will provide beans that Spring Data Elasticsearch has not yet provided.

For the use of http clients that means, that a custom client would need to be created in Spring Data Elasticsearch and be provided as a bean (which definitely is possible) and then it could be used by other components as well. Spring Data Elasticsearch cannot use beans provided by Spring Boot, Boot beans are created/initialized after Spring Data Elasticsearch beans.

And I did not yet check what clients will be supported, with a first glance I did not see that Webflux webclient is supported, and even if it were supported as a client, please note, that this would not provide a reactive client handling at all, for this all the calls in the Elasticsearch client API need reactive versions.

But I'll have a look of what can be used for Spring Data Elasticsearch of this.

[tongdaoqa]

Thank you

[patpatpat123]

I also have a

 @Bean
    public WebClient getWebClient(WebClient.Builder builder) {
       //web client which has tracing, logging, time out, etc...
    }

In which I would really hope spring data elasticsearch can leverage.

Thank you in advanced @sothawo if you can help on this

[sothawo]

This seems not to be part of the current version 8.8.1 of the Elasticsearch client.

[patpatpat123]

It is now available with the latest 8.9!

[patpatpat123]

Saw the integration with version 8.9, thank you again @sothawo

Is this feature "doable"?

[sothawo]

Might be. The official docs do not mention or explain this, so it would be down to go into the implementation and check what can be done.

[patpatpat123]

I am also reading the code, and trying to see how to integrate with WebClient from Spring Data Elasticsearch.
This is quite an interesting feature.
Unfortunately, not super familiar with this repo, it is a bit hard.

I saw those:
https://github.com/elastic/elasticsearch-java/tree/main/example-transports
https://github.com/elastic/elasticsearch-java/tree/main/example-transports/src/main/java/co/elastic/clients/transport/netty

Do you think those examples are helpful?

[patpatpat123]

Any luck on this please?

[sothawo]

With #2702 it is now possible to provide a custom ElasticsearchTransport implementation in an implementation of the ElasticsearchConfiguration class, that opens the door to use other clients by providing a transport for these clients.

[patpatpat123]

Thanks! let me try to pass the WebClient to this.

[brunopereira7]

Hi @patpatpat123, any luck on integrating it with WebClient?

[KafkaProServerless]

@sothawo

Upvoting this.
Having an existing WebClient bean is very common.

Could we please have an example of an integration with existing WebClient please?
Thank you

[sothawo]

Sorry, I don't have the time to write another ElasticsearchTransport implementation that uses WebClient and it would bring no additional value to Spring Data Elasticsearch itself.

It would not give you a true reactive client to access Elasticsearchanyway, as it would be just plugged into the normal Transport interface which has no reactive methods at all, only the imperative and async ones which we already use for the reactive implementation.