HDDS-10749. Shutdown datanode when RatisServer is down #6587
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
A normal DN shutdown log, first XceiverServerRatis is stopped, "Stopping XceiverServerRatis 01effdc6-dad1-4bf3-916a-749d9aa7e5e5", then ContainerStateMachine is stopped, "Stopping ContainerStateMachine for group-5EA60976374E". |
|
A DN shutdown due to Ratis server is shutdown. First ContainerStateMachine is closed, "Container statemachine is closed by ratis, terminating HddsDatanodeService", then XceiverServerRatis is stopped, "Stopping XceiverServerRatis 01effdc6-dad1-4bf3-916a-749d9aa7e5e5". |
smengcl
reviewed
Apr 26, 2024
Comment on lines
+1143
to
+1145
| // wait a while for other pipeline's ContainerStateMachine.close() called. | ||
| try { | ||
| Thread.sleep(10000); |
There was a problem hiding this comment.
Is there a more reliable way to wait for other pipeline closure other than sleep here?
And what happens if there are still unclosed pipeline after 10 seconds' wait?
There was a problem hiding this comment.
Here the ContainerStateMachine.java, which is called by each pipeline, it doesn't have the knowledge of other pipelines. 10s here is try to let other pipelines have time to close. And it's only memory operation in ContainerStateMachine.close() call. Missed a call to ContainerStateMachine.close() is not a big issue. So Shutdown immediately, wait 5s or 10s, has no big difference. Just think wait a while would be better, like when executor pool is shutdown.
adoroszlai
requested changes
Apr 29, 2024
There was a problem hiding this comment.
Thanks @ChenSammi for working on this.
- Each Raft group, which corresponds to a pipeline membership in Datanode, has its own
ContainerStateMachine. - When SCM detects a dead datanode, it closes all pipelines associated with it, which triggers close of the state machine in the other two nodes.
So with this patch stopping a datanode or closing a pipeline kills other datanodes.
Note that due with multi-Raft, the effect can be cascading, since datanodes may be associated with different sets of other nodes for each Raft group.
Repro:
cd hadoop-ozone/dist/target/ozone-1.5.0-SNAPSHOT/compose/ozone
OZONE_DATANODES=6 ./run.sh -d
docker-compose exec scm ozone admin safemode wait -t 60
docker-compose ps
docker-compose up -d --no-recreate --scale datanode=5
docker-compose ps
sleep 120
docker-compose ps
Datanodes at the last step:
Name Command State Ports
-----------------------------------------------------------------------------------------------------------------------------------------------------
ozone_datanode_1 /usr/local/bin/dumb-init - ... Up 0.0.0.0:33008->19864/tcp,:::33008->19864/tcp, 0.0.0.0:33011->9882/tcp,:::33011->9882/tcp
ozone_datanode_2 /usr/local/bin/dumb-init - ... Exit 1
ozone_datanode_3 /usr/local/bin/dumb-init - ... Up 0.0.0.0:33014->19864/tcp,:::33014->19864/tcp, 0.0.0.0:33015->9882/tcp,:::33015->9882/tcp
ozone_datanode_4 /usr/local/bin/dumb-init - ... Up 0.0.0.0:33006->19864/tcp,:::33006->19864/tcp, 0.0.0.0:33007->9882/tcp,:::33007->9882/tcp
ozone_datanode_5 /usr/local/bin/dumb-init - ... Exit 1
|
@adoroszlai , I noticed the impact to the integration test too. It looks like terminate the DN in ContainerStateMachine is not a good idea for DN. Let me think if there is other solutions. |
|
Wait for RATIS release including https://issues.apache.org/jira/browse/RATIS-2066. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Currently, when RatisServer is down(mainly due to long GC which exceeds the ratis close threshold), Datanode is still running and in HEALTHY and IN_SERVICE state, which is confusing.
This tasks will shutdown the Datanode after RatisServer is down.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-10749
How was this patch tested?
Manual test