Indices.UpdateAliases() not working as expected

[plibauer]

Elastic.Clients.Elasticsearch version: 8.11

Elasticsearch version: 8.3.3

.NET runtime version: 4.8.09032

Operating system version: Windows 11

Description of the problem including expected versus actual behavior:
When attempting an atomic update of an index alias to remove the old write index and add a new write index, the update fails and reports that there can only be one configured is_write_index. If the update is done using a raw JSON request, it works as expected.

        string oldWriteIndex = "test_bulk_index-000001";
        string newWriteIndex = "test_bulk_index-000002";
        string alias = "test_bulk_index";

        var updResp = client.Indices.UpdateAliases(a => a
           .Actions(b => b
               .Add(o => o
                   .Indices(oldWriteIndex)
                   .Alias(alias)
                   .IsWriteIndex(false))
               .Add(n => n
                   .Indices(newWriteIndex)
                   .Alias(alias)
                   .IsWriteIndex(true))
               ));

A workaround for this is to use a raw JSON request and target the "_aliases" endpoint directly using the client.Transport.Request() method.

{
"actions": [
{
"add": {
"index": "test_bulk_index-000001",
"alias": "test_bulk_index",
"is_write_index": false
}
},
{
"add": {
"index": "test_bulk_index-000002",
"alias": "test_bulk_index",
"is_write_index": true
}
}
]
}

Steps to reproduce:

1. Set up an initial index with an associated alias and make it the write index.
2. Create a second index which is to be the new write index.
3. Call the Indices.UpdateAliases() code as above to try to atomically update the current is_write_index to the latest index

Expected behavior

The expected behaviour here is that the old index will have is_write_index set to false, followed by the new index having it set to true and any current indexing that uses the alias to seamlessly transition to updating the new index, ie. it should be an atomic operation.

Provide ConnectionSettings (if relevant):

Provide DebugInformation (if relevant):

Invalid Elasticsearch response built from a unsuccessful (500) low level call on POST: /_aliases?pretty=true&error_trace=true
Exception: The remote server returned an error: (500) Internal Server Error. Call: Status code 500 from: POST /_aliases?pretty=true&error_trace=true. ServerError: Type: illegal_state_exception Reason: "alias [test_bulk_index] has more than one write index [test_bulk_index-000001,test_bulk_index-000002]"

Audit trail of this API call:

• [1] BadResponse: Node: http://localhost:9200/ Took: 00:00:00.0180064

OriginalException: Elastic.Transport.TransportException: The remote server returned an error: (500) Internal Server Error. Call: Status code 500 from: POST /_aliases?pretty=true&error_trace=true. ServerError: Type: illegal_state_exception Reason: "alias [test_bulk_index] has more than one write index [test_bulk_index-000001,test_bulk_index-000002]" ---> System.Net.WebException: The remote server returned an error: (500) Internal Server Error.

at System.Net.HttpWebRequest.GetResponse()
at Elastic.Transport.HttpWebRequestTransportClient.d__8`1.MoveNext() in /home/runner/work/elastic-transport-net/elastic-transport-net/src/Elastic.Transport/Components/TransportClient/HttpWebRequestTransportClient.cs:line 142
--- End of inner exception stack trace ---

Request:

{"actions":[{"add":{"alias":"test_bulk_index","indices":"test_bulk_index-000002","is_write_index":true}}]}

Response:

{
"error" : {
"root_cause" : [
{
"type" : "illegal_state_exception",
"reason" : "alias [test_bulk_index] has more than one write index [test_bulk_index-000001,test_bulk_index-000002]",
"stack_trace" : "org.elasticsearch.ElasticsearchException$1: alias [test_bulk_index] has more than one write index [test_bulk_index-000001,test_bulk_index-000002]\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:638)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:566)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.rest.BytesRestResponse.build(BytesRestResponse.java:145)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.rest.BytesRestResponse.(BytesRestResponse.java:101)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.rest.BytesRestResponse.(BytesRestResponse.java:81)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:55)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.client.internal.node.NodeClient$ActionResponseTaskListener.onFailure(NodeClient.java:185)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.tasks.TaskManager$1.onFailure(TaskManager.java:180)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$2(TransportMasterNodeAction.java:212)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.action.ActionListener$DelegatingActionListener.onFailure(ActionListener.java:218)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.action.admin.indices.alias.TransportIndicesAliasesAction.lambda$masterOperation$2(TransportIndicesAliasesAction.java:243)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.action.ActionListener$DelegatingActionListener.onFailure(ActionListener.java:218)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.AckedClusterStateUpdateTask.onFailure(AckedClusterStateUpdateTask.java:75)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.service.MasterService$Batcher$UpdateTask.onFailure(MasterService.java:184)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:253)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:156)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:110)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:148)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:710)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:260)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:223)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\r\n\tat java.base/java.lang.Thread.run(Thread.java:833)\r\nCaused by: java.lang.IllegalStateException: alias [test_bulk_index] has more than one write index [test_bulk_index-000001,test_bulk_index-000002]\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.metadata.Metadata$Builder.validateAlias(Metadata.java:2042)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.metadata.Metadata$Builder.build(Metadata.java:1786)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.ClusterState$Builder.metadata(ClusterState.java:658)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.metadata.MetadataIndexAliasesService.applyAliasActions(MetadataIndexAliasesService.java:186)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.metadata.MetadataIndexAliasesService$1.execute(MetadataIndexAliasesService.java:75)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.service.MasterService$UnbatchedExecutor.execute(MasterService.java:482)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:908)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:878)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:248)\r\n\t... 9 more\r\n"
}
],
"type" : "illegal_state_exception",
"reason" : "alias [test_bulk_index] has more than one write index [test_bulk_index-000001,test_bulk_index-000002]",
"stack_trace" : "java.lang.IllegalStateException: alias [test_bulk_index] has more than one write index [test_bulk_index-000001,test_bulk_index-000002]\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.metadata.Metadata$Builder.validateAlias(Metadata.java:2042)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.metadata.Metadata$Builder.build(Metadata.java:1786)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.ClusterState$Builder.metadata(ClusterState.java:658)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.metadata.MetadataIndexAliasesService.applyAliasActions(MetadataIndexAliasesService.java:186)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.metadata.MetadataIndexAliasesService$1.execute(MetadataIndexAliasesService.java:75)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.service.MasterService$UnbatchedExecutor.execute(MasterService.java:482)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:908)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:878)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:248)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:156)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:110)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:148)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:710)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:260)\r\n\tat org.elasticsearch.server@8.3.3/org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:223)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\r\n\tat java.base/java.lang.Thread.run(Thread.java:833)\r\n"
},
"status" : 500
}

[flobernd]

Hi, thanks for reporting. Based on the request payload, it seems like the second action is serialized but the first one is missing.

[flobernd]

Hi @plibauer,

it seems like this is not an issue in the library. It works fine in 8.13.7, if you change the call like this:

var updResp = await client.Indices.UpdateAliasesAsync<Person>(a => a
    .Actions(
        [
            b => b.Add(o => o
                .Indices(oldWriteIndex)
                .Alias(alias)
                .IsWriteIndex(false)),
            b => b.Add(n => n
                .Indices(newWriteIndex)
                .Alias(alias)
                .IsWriteIndex(true))
        ]
    ));

Actions has an overload that accepts an params array of actions. This is the correct one to use. Using .Add multiple times will otherwise override the previous values. This is a little bit unintuitive as the .Add action implies that it adds something to a list. In this case it means that the descriptor gets confugired to use an AddAction.