[FEATURE]: New AWS opensearch request for OSS

[Billionrafiya]

After upgrading to 3.18.0 we have seen couple of issues:

1. After upgrading conductor server in my local and this is critically important for our business requirement as well for production.
   Noticed that v3.18.0 is dependent on Elastic Search 7 version. As currently we have Elastic search 6.8 version and which is breaking our code and we would be seeing the downtime/errors related to restClientBuilder if we upgrade conductor with latest version and with Elastic search 6.8 in production and non prod environments. Could you please let us know whether this was supposed to be using Elastic Search version as 7 or higher, or is the bug/dependency which is introduced in this v3.18.0 version.

OSS comuunity team has told that ES7 is the default indexing for the server going forward.

Slack channel where the discussion happened and OSS replied.

https://orkes-conductor.slack.com/archives/C02KJ820XPW/p1711986244389439

2. After upgrading ES to 7.10 we saw that Netfix conductor was failing with query failed and compatbility issues with ES and Netflix conductor. Error attached for the same.

The problem is rest client version is 7.14 while ES version is 7.10, which is causing issue.
Is there any version of ES avl in AWS which is >= 7.14

if (major == 6 || (major == 7 && minor < 14)) {
if ("You Know, for Search".equalsIgnoreCase(mainResponse.getTagline()) == false) {
return Optional.of("Invalid or missing tagline [" + mainResponse.getTagline() + "]");
} if (major == 7) {
// >= 7.0 and < 7.14
String responseFlavor = mainResponse.getVersion().getBuildFlavor();
if ("default".equals(responseFlavor) == false) {
// Flavor is unknown when running tests, and non-mocked responses will return an unknown flavor
if (Build.CURRENT.flavor() != Build.Flavor.UNKNOWN || "unknown".equals(responseFlavor) == false) {
return Optional.of("Invalid or missing build flavor [" + responseFlavor + "]");
}
}
} return Optional.empty();
}

2024-05-06T14:11:51.294Z ERROR 1 --- [conductor] [m-task-worker-2] c.n.c.e.d.i.ElasticSearchRestDAOV7 : Failed to index org.elasticsearch.action.bulk.BulkRequest/unset for request type: task
org.elasticsearch.ElasticsearchException: Invalid or missing build flavor [oss]
at org.elasticsearch.client.RestHighLevelClient.performClientRequest(RestHighLevelClient.java:2701) ~[elasticsearch-rest-high-level-client-7.17.16.jar:7.17.16]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2171) ~[elasticsearch-rest-high-level-client-7.17.16.jar:7.17.16]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:2137) ~[elasticsearch-rest-high-level-client-7.17.16.jar:7.17.16]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:2105) ~[elasticsearch-rest-high-level-client-7.17.16.jar:7.17.16]
at org.elasticsearch.client.RestHighLevelClient.bulk(RestHighLevelClient.java:620) ~[elasticsearch-rest-high-level-client-7.17.16.jar:7.17.16]
at com.netflix.conductor.es7.dao.index.ElasticSearchRestDAOV7.lambda$indexWithRetry$18(ElasticSearchRestDAOV7.java:1286) ~[conductor-es7-persistence-3.18.0.jar:?]
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:335) ~[spring-retry-2.0.5.jar:?]
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:211) ~[spring-retry-2.0.5.jar:?]
at com.netflix.conductor.es7.dao.index.ElasticSearchRestDAOV7.indexWithRetry(ElasticSearchRestDAOV7.java:1285) ~[conductor-es7-persistence-3.18.0.jar:?]
at com.netflix.conductor.es7.dao.index.ElasticSearchRestDAOV7.indexBulkRequest(ElasticSearchRestDAOV7.java:1265) ~[conductor-es7-persistence-3.18.0.jar:?]
at com.netflix.conductor.es7.dao.index.ElasticSearchRestDAOV7.indexObject(ElasticSearchRestDAOV7.java:1257) ~[conductor-es7-persistence-3.18.0.jar:?]
at com.netflix.conductor.es7.dao.index.ElasticSearchRestDAOV7.indexTask(ElasticSearchRestDAOV7.java:522) ~[conductor-es7-persistence-3.18.0.jar:?]
at com.netflix.conductor.core.dal.ExecutionDAOFacade.updateTask(ExecutionDAOFacade.java:517) ~[conductor-core-3.18.0.jar:?]
at java.base/java.lang.Iterable.forEach(Unknown Source) ~[?:?]

3.Update from OOS team was the High Level Client is guaranteed to be able to communicate with any Elasticsearch node running on the same major version and greater or equal minor version. It doesn’t need to be in the same minor version as the Elasticsearch nodes it communicates with, as it is forward compatible meaning that it supports communicating with later versions of Elasticsearch than the one it was developed for you might need to build conductor with your own ES7 client version or upgrade elastic.

https://orkes-conductor.slack.com/archives/C02KJ820XPW/p1715038099019499?thread_ts=1715010886.733999&cid=C02KJ820XPW

Upon checking we have observed that AWS doesnt support and we can not go to 7.15. Either we have to go to AWS opensearch.
Could you please udpate the OSS code with Opensearch.

4. Another issue, we were moving away from Elasticsearch to Postgresql for indexing, while doing so we are facing one issue, where we are not able to search the workflows which we already indexed in Elasticsearch.

Update from viren, you will have to re-index them in postgres if you want to do that. Since the system now sends all the queries to postgres if you switch to PG from ES. There is no out of box support to do this however, This is challenging part in production and we cant go ahead.

[missedone]

i can't speak for all the issue you mentioned, but regards to

After upgrading ES to 7.10 we saw that Netfix conductor was failing with query failed and compatbility issues with ES and Netflix conductor. Error attached for the same.

we had the same issue, the solution is to pin the es client version to 7.10.x, the implementation depends on how you use conductor: build from source vs import as dependencies to your project.