Fix various bugs related to restart of StatefulSet pods that may result in connectivity issues #31605
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
christarazi
added
kind/bug
This comment was marked as outdated.
This comment was marked as outdated.
2 tasks
christarazi
force-pushed
the
pr/christarazi/fix-sts
branch
from
b8e3372
to
f806341
Compare
This comment was marked as outdated.
This comment was marked as outdated.
christarazi
commented
Mar 28, 2024
christarazi
changed the title
christarazi
requested review from
joamaki,
tklauser,
squeed,
marseel and
learnitall
christarazi
force-pushed
the
pr/christarazi/fix-sts
branch
from
f806341
to
ad757a8
Compare
|
Is this fix something we want to backport to 1.15? From the linked issue it seems all currently supported stable versions are affected. |
This commit causes the Kubernetes Pod watcher to trigger endpoint identity updates if the Pod UID changes. This covers the third scenario pointed out in [1], see issue for more details. To summarize, with StatefulSet restarts, it is possible for the apiserver to coalesce Pod events where the delete and add event are replaced with a single update event that includes, for example a label change. If Cilium indexes based on namespace/name of the pod, then it will miss this update event and therefore, Cilium must use the UID of the pod to disambiguate. [1]: cilium#30409 Signed-off-by: Chris Tarazi <chris@isovalent.com>
christarazi
force-pushed
the
pr/christarazi/fix-sts
branch
from
0dc7a82
to
a55ca77
Compare
This comment was marked as outdated.
This comment was marked as outdated.
christarazi
force-pushed
the
pr/christarazi/fix-sts
branch
2 times, most recently
from
fda9152
to
b56ecff
Compare
This comment was marked as outdated.
This comment was marked as outdated.
squeed
approved these changes
May 8, 2024
Building off of the previous commit, This commit modifies if the CNI ADD handling logic to pass the Kubernetes Pod UID in the ADD request to the Agent. In addition, a extra condition checks whether the UID of the pod corresponding to the CNI ADD request is the same UID which Cilium is aware of (within its pod informer store) during endpoint creation. This is key for handling StatefulSets properly, see [1]. If the UID is outdated, this means that Cilium has not yet received the updated pod event that corresponds with the CNI ADD. In this case, Cilium will attempt to query the pod store up for to 2 seconds. If the pod store does not have the altest pod reference after 2 seconds, Cilium will trigger a direct fetch from the apiserver, as to avoid a potentially large window of time where an endpoint cannot be created. If the direct fetch fails, the endpoint will be created with the 'init' identity until the metadata resolver (controller) is able to fetch the latest pod metadata. The reason for these changes is due to the way StatefulSets are implemented in Kubernetes. One significant property of StatefulSets is that they have a fixed name. When a StatefulSet is restarted, Cilium is unable to properly handle it because Cilium indexes based on namespace/name[1]. Therefore, we must pass a UID through the CNI ADD in order to ensure that when Cilium creates a new endpoint, it is created with the most up-to-date corresponding Kubernetes metadata. [1]: cilium#30409, see case 1. Signed-off-by: Chris Tarazi <chris@isovalent.com>
Previously, the logic in the pod watcher did not properly check for changes in a pod's IPs, incorrectly relying on the length of the IP strings changing, rather than the contents. This commit fixes that while also simplifying the logic into a simpler one-liner. Reported-by: Casey Callendrello <cdc@isovalent.com> Signed-off-by: Chris Tarazi <chris@isovalent.com>
christarazi
force-pushed
the
pr/christarazi/fix-sts
branch
from
b56ecff
to
f56b7ee
Compare
|
/test |
christarazi
added
backport-pending/1.15
maintainer-s-little-helper
bot
moved this from Needs backport from main
to Backport pending to v1.15
in 1.15.5
|
Is this being backported to 1.14 ? 🙏 |
github-actions
bot
added
backport-done/1.15
|
@christarazi sorry for the ping , i just wanted to check if there is any plan of porting this fix to 1.14 ? thanks 🙏 |
|
@primeroz Right now only v1.15 backport was considered due to the nature of the changes being non-trivial. The risk of backporting to older stable versions is that we introduce a regression in very stable versions of Cilium. |
|
@christarazi i understand and we will check internally if is worth for us to upgrade to 1.15 Just to confirm though the issue is present in 1.14 as well right ? ( we think we did hit it ) thanks 🙏 |
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes: #30409