search-backend-module-elasticsearch: ensure all stale indices are deleted

[tcardonne]

Hey, I just made a Pull Request!

We ran into a case where the indexer would fail but leave a half-filled index without deleting it (#24680 should allow us to understand why).
As indices are not deleted, we're exhausting the allowed number of shards on our OpenSearch cluster.

This PR changes the way stale indices are cleaned. Instead of relying on aliases, we simply list indices that match the indexer's pattern (some-type-index__*) and delete them after indexation.

I've marked the change as breaking because someone may use some-type-index__some-index-that-shouldnt-be-deleted as an index name (not managed by Backstage Search).
However the wildcard pattern is already used to remove aliases, so IMHO we can commit to use the pattern to detect and delete stale indices.

✔️ Checklist

• A changeset describing the change and affected packages. (more info)
• Added or updated documentation
• Tests for new functionality and regression tests for bug fixes
• Screenshots attached (for UI changes)
• All your commits have a Signed-off-by line in the message. (more info)

[backstage-goalie]

Changed Packages

Package Name	Package Path	Changeset Bump	Current Version
@backstage/plugin-search-backend-module-elasticsearch	plugins/search-backend-module-elasticsearch	minor	v1.4.2-next.0

Need to mark alias function as deprecated

[tcardonne]

Hi @djamaile, thanks for the review!

I've re-added getAliases on ElasticSearchClientWrapper and marked it as deprecated. Not sure about the deprecation message, hope it works this way 🤞

[djamaile]

Is there an action that the adopter should take before upgrading to this new version? For example, updating their index patterns 🤔

[tcardonne]

I've added an action to verify that the pattern isn't conflicting with other indices not managed by Backstage
(which should already be an "issue" since the indexer removes aliases on these indices)

[backstage-goalie]

Thanks for the contribution!
All commits need to be DCO signed before they are reviewed. Please refer to the the DCO section in CONTRIBUTING.md or the DCO status for more info.

[tcardonne]

Hi @djamaile,

I've updated the PR to handle deletion in chunks of 50 indices to fix an issue that can occur when having a lot of stale indices to delete.

The indices delete method in ES/OS clients generate a DELETE /index1,index2,index3.. request, and since the size of an URL is limited, it fails to send the delete request for ~200-300 indices (depending on the pattern size).
To mitigate this I'm splitting the removableIndices array in chunks of 50 indices.

Let me know if something's needed to move this forward!