-
Notifications
You must be signed in to change notification settings - Fork 24.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[CI] SearchableSnapshotActionIT testSearchableSnapshotsInHotPhasePinnedToHotNodes failing #107421
Comments
Pinging @elastic/es-distributed (Team:Distributed) |
The failure reported above concerns ILM, In particular the test creates a data stream (with backing index I am re-labeling to data management to check why. |
Pinging @elastic/es-data-management (Team:Data Management) |
Leaving some notes here for when I get back to this:
Conclusion so far: ILM fired a rollover request, which created a new index, but then a second rollover request got executed before the callback of the first rollover request was completed. |
Ah after reading through my notes again (I am a rubber duck?), I'm guessing the first rollover is from ILM and the second (erroneous) one is from Line 860 in 25e3551
I'll try to reproduce this later today. That seems like the most plausible explanation right now. Although, I would have expected this to happen more often. |
Manually rolling over the data stream was 1. not necessary, and 2. causing issues when it happened after ILM had already rolled over, which would make ILM convert the second backing index into a searchable snapshot as well, possibly preventing the data stream from being deleted. Fixes elastic#107421
I wasn't able to reproduce this right away, as the timing has to be just right in order for this to happen, but I'm fairly confident that #108790 will prevent this failure from happening again. |
Build scan:
https://gradle-enterprise.elastic.co/s/unsdhiimapjfc/tests/:x-pack:plugin:ilm:qa:multi-node:javaRestTest/org.elasticsearch.xpack.ilm.actions.SearchableSnapshotActionIT/testSearchableSnapshotsInHotPhasePinnedToHotNodes
Reproduction line:
Applicable branches:
main
Reproduces locally?:
No
Failure history:
Failure dashboard for
org.elasticsearch.xpack.ilm.actions.SearchableSnapshotActionIT#testSearchableSnapshotsInHotPhasePinnedToHotNodes
Failure excerpt:
The text was updated successfully, but these errors were encountered: