Sticky IPs for StatefulSet

[bprashanth]

Most of the databases I (https://github.com/kubernetes/kubernetes/tree/master/test/e2e/testing-manifests/petset) and others (#28718 (comment)) have prototyped seem to handle DNS properly, but there are murmurs that some do not (#23828 (comment)), and dont' have plans to do so (#28718 (comment)).

I'd still vote for defering any implementation till we have the end to end models fleshed out. One might imagine that databases understand the importance of DNS ttl.

[chrislovecnm]

I am checking to see if Cassandra can run on DNS fully. It works for lookup of seeds, but I am seeing that IP Addresses gets bound to a token range.

[chrislovecnm]

I have confirmed that Cassandra does need this. The Datastax team has toyed with the idea of using DNS, but at this time it is not support to use pure DNS with Cassandra.

[bprashanth]

Great! do you have a bug / doc for context

[chrislovecnm]

@bprashanth nope email from Datastax folks. They asked me to file a Jira if I wanted to recommend a change.

[zefciu]

I have an idea to implement this feature in calico-containers. I believe that a thing like sticky IPs is not something k8s is responsible for. I still however need a confirmation that this is a requirement for galera to get a blessing for this feature.

[chrislovecnm]

@zefciu with calico would this be able to work with internal IPs / DNS that PetSet uses? We need a pet to have a sticky IP address. Also I understand using calico, but this may want to be self contained in k8s.

[chrislovecnm]

To add more color sticky IP address that is in the internal private subnet the cluster and minions use. This is not a sticky public ip.

[zefciu]

The solution is to use annotations, that sent by CNI would make calico either use dynamic or static IPs. I don't know how can we solve the static IP logic in k8s itself if its outsourcing all the job of setting up network interfaces to plugins.

[bprashanth]

The way I would like to solve it is by defining a staitc-ip-subnet (just like podCIDR assigned to nodes), from which we'd draw these limited ips and assign them to pods. If a pod with the limited edition ip dies, we reprogram the routes so traffic flows to whichever node it lands, just like we setup routes today to route podCIDR to specific nodes.

The easier way to solve static ip is through a Service vip, but I don't like that for a couple of reasons (occupies iptables space, requires Svc per pod, wont work cross kube-cluster).

[zefciu]

@bprashanth: and how would your solution work with plugins? Would it simply sent this desired IP to the plugin, or will it take some of the plugin's responsibilities?

[bprashanth]

network plugins are responsible for allocating ips from a given range today, not between nodes. This range is the pod cidr. Something assigns podCIDRs and setups up routing (whatever that may be, it isn't a plugin yet. It could be CP specific route controller or something like flannel). The plugin will only be responsible for eg: creating a veth with the allocated ip and shoving in the netns. IPAM itself is a plugin within the CNI plugin.

[magicwang-cn]

init-containers also shares the same network with the whole pod, so why not get the sticky ips in the init-containers?

[slaskawi]

As for JBoss projects based on JGroups (like Infinispan for example), we probably need to write a new discovery protocol based on DNS (I proposed it on Infinispan dev mailing list and waiting for a response). Currently some of us use KUBE_PING (which queries Kubernetes API and collects containers) but trusting DNS would probably be a much better option.

However we (the Infinispan Team) would be very interested in exposing PetSets to the outside world. Our Hot Rod client can take the advantage of topology information and optimize queries. Having a public Sticky IPs (or anything that let's client decide to which Pod the request should be forwarded) would be very important for us.

[chrislovecnm]

@thockin this is the one I meant to comment on. Who is setting the priority of this one? Would a proposal be a good start?

[thockin]

A proposal is always a good start

[krmayankk]

@chrislovecnm are you writing this proposal ?

[chrislovecnm]

@krmayankk not had any time ... And frankly found a work around ish for Cassandra

[k8s-github-robot]

@bprashanth There are no sig labels on this issue. Please add a sig label by:
(1) mentioning a sig: @kubernetes/sig-<team-name>-misc
(2) specifying the label manually: /sig <label>

Note: method (1) will trigger a notification to the team. You can find the team list here.

[caseydavenport]

/sig network

[braedon]

Hi @chrislovecnm, we've having problems running Cassandra in stateful sets due to this - while we're waiting for a solution, could you share your workaround? Thanks!

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[agolomoodysaada]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[aspyct]

/remove-lifecycle stale

[vadalikrishna]

@krmayankk not had any time ... And frankly found a work around ish for Cassandra

Can you please share your work around, I am facing a similar issue with cassandra in statefulset.

[cscetbon]

@vadalikrishna @krmayankk any information on the workaround would be much appreciated

[madireddyr]

@krmayankk @vadalikrishna chrislovecnm : any info how you resolved this issue

[cscetbon]

@brouberol What if two cassandra pods are going down at the same time?

Say pod A is running on k8s node 1, and pod B is running on k8s node 2.
They both go down at the same time.
What's to prevent pod A from starting again on node 2, and therefore getting the IP address that was previously assigned to pod B?
It's definitely a problem when both nodes are getting IPs of each other. Cassandra complains that an existing node already holds the tokens the current node is trying to get and refuses to start.

That's why if someone like @chrislovecnm has a workaround it would be much appreciated to know it.

@allamand Does someone have tested the option from calico :

annotations:
"cni.projectcalico.org/ipAddrs": "["192.168.0.1"]"

as you know, the issue is we can't assign different annotations to nodes belonging to a StatefulSet

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[dpepper]

Say pod A is running on k8s node 1, and pod B is running on k8s node 2.
They both go down at the same time.
What's to prevent pod A from starting again on node 2, and therefore getting the IP address that was previously assigned to pod B?
It's definitely a problem when both nodes are getting IPs of each other. Cassandra complains that an existing node already holds the tokens the current node is trying to get and refuses to start.

Did anyone find a solution for this?

[allamand]

You can force podA to stay on node1 using local storage and persistent volume claim

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[arianvp]

Can we reopen this? This is still very relevant for many workloads that now have issues running on kubernetes manually. Including Audio/Video (STUN/TURN etc), redis and cassandra.

[arianvp]

If you are using Calico, you could add https://docs.tigera.io/networking/use-specific-ip to each pod manually. To assign a fixed IP to each pod. However you'd have to add the annotation to new pods that get created when you scale up; which is a bit annoying.

Also it's racey; as the Pod is first created with a different IP; and then you change it after the fact; which is not ideal as some workloads will get very upset from that (e.g. cassandra) which is what we're trying to prevent with this issue in the first place. No. you cabn only set these fields during pod creation time unfortunately

It would be really cool if we could add an annotation to the StatefulSet that Calico interprets and then does this automatically. Listen to pods being created; and then add a fixed IP to the pod

[alitoufighi]

Can we have this issue reopened?