Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CURATOR-696. Fix double leader for LeaderLatch #500

Merged
merged 4 commits into from
May 21, 2024

Conversation

tisonkun
Copy link
Member

@tisonkun tisonkun commented May 10, 2024

A possible fix to CURATOR-696.

I'm thinking whether this has more corner cases ...

cc @gianm @kezhuw @eolivelli

Signed-off-by: tison <wander4096@gmail.com>
Comment on lines +559 to 569
final long ephemeralOwner =
event.getStat() != null ? event.getStat().getEphemeralOwner() : -1;
final long thisSessionId =
client.getZookeeperClient().getZooKeeper().getSessionId();
if (ephemeralOwner != thisSessionId) {
// this node is gone - reset
reset();
} else {
lastPathIsLeader.set(localOurPath);
setLeadership(true);
}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the major fix.

Copy link
Member

@asdf2014 asdf2014 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for the quick fix

@tisonkun tisonkun marked this pull request as draft May 10, 2024 10:44
@tisonkun

This comment was marked as outdated.

Signed-off-by: tison <wander4096@gmail.com>
@tisonkun tisonkun marked this pull request as ready for review May 10, 2024 10:54
@kezhuw kezhuw self-requested a review May 10, 2024 11:31
Copy link
Member

@kezhuw kezhuw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

I think we can also set up a watcher for ourPath to gain ability to "perceive" external deletion (from human or bugs 😮‍💨 ?). It is separated anyway.

@tisonkun
Copy link
Member Author

tisonkun commented May 12, 2024

@kezhuw Thanks for your review.

a watcher for ourPath to gain ability to "perceive" external deletion

Curator has a basic assumption that we own the node (TN-7). This patch fixes a bug that we don't manage the node (ephemeral owner, a.k.a, session id) properly. We may not handle "external deletion".

And such a watcher doesn't fix our issue here, because it can take some time the deletion pass to the client before it gives up the leadership.

@tisonkun tisonkun requested a review from eolivelli May 12, 2024 11:34
Copy link
Contributor

@eolivelli eolivelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think that it is possible to add a test case?

Signed-off-by: tison <wander4096@gmail.com>
@tisonkun
Copy link
Member Author

@eolivelli pushed. Ensure that it fails without this patch and passes with this patch.

@tisonkun
Copy link
Member Author

image

Reproduced with a more handy tests. The test pushed above can prevent regression already.

@tisonkun tisonkun requested a review from eolivelli May 13, 2024 06:44
Copy link
Member

@kezhuw kezhuw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the hard work! A minor suggestion left.

@tisonkun
Copy link
Member Author

I'll merge this patch later this week if no more inputs.

I'd still seek possibility to reduce the test consumption since it now takes 140+ minutes to finish ...

@tisonkun tisonkun merged commit 1027c2c into apache:master May 21, 2024
6 checks passed
@tisonkun tisonkun deleted the CURATOR-696 branch May 21, 2024 14:59
@razinbouzar
Copy link

@tisonkun do you know when the 5.7.0 release will be made available?

@tisonkun
Copy link
Member Author

@razinbouzar I'll start the discussion today and hopefully vote in two to three weeks. You can keep an eye on the dev mailing list (dev@curator.apache.org).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
5 participants