Proposal for implementing nominal services AKA StatefulSets AKA The-Proposal-Formerly-Known-As-PetSets #18016

smarterclayton · 2015-12-01T13:26:53Z

This is the draft proposal for #260.

This change is

davidopp · 2015-12-01T21:42:21Z

docs/proposals/petset.md

+Like a replication controller, a PetSet may be targeted by an autoscaler. The PetSet makes no assumptions
+about upgrading or altering the pods in the set (similar to a DaemonSet) - instead, the user can trigger
+graceful deletion and the PetSet will replace the terminated member with the newer template once it exits.
+Future proposals may offer update capabilities. A PetSet requires RunAlways pods.


I think you mean RestartPolicyAlways (or "a restart policy of Always")

davidopp · 2015-12-01T22:00:18Z

I like this design a lot. I think this is going to make a lot of users very happy, and will make it practical for normal users to deploy applications that today require bizarre contortions (like creating one controller per pod).

eswarbala · 2015-12-02T03:27:34Z

Nice proposal and looking forward to this! I am one such user looking forward to move away from one RC per instance to achieve this.

ant31 · 2015-12-02T12:30:37Z

Active-active
*Need active-active master example - galera? *

I propose RabbitMQ as active-active exemple. It also has some specific requirement being master-master.
i.e: When the entire cluster is brought down, the last node to go down must be the first node to be brought online.

ant31 · 2015-12-02T13:08:20Z

Instances can migrate from machine to machine as necessary and are not tied to an instance

This requires to use a NetworkStorage, I would prefer to have the choice to allow or not my Pets to be re-scheduled on an another node. If this is possible, using local storages on the cluster would be easy.
A simple example is when someone simply wants to mount a volume like `hostPath: /mnt/mypet-$IDENTITY'.

ncdc · 2015-12-02T14:37:54Z

cc @kubernetes/rh-cluster-infra @kubernetes/rh-scalability

smarterclayton · 2015-12-02T14:43:55Z

Thanks, will add rabbit as an example.

On Dec 2, 2015, at 7:30 AM, Antoine Legrand notifications@github.com
wrote:

Active-active
*Need active-active master example - galera? *

I propose RabbitMQ as active-active exemple. It also has some specific
requirement being master-master.
i.e: When the entire cluster is brought down, the last node to go down must
be the first node to be brought online.

—
Reply to this email directly or view it on GitHub
#18016 (comment)
.

smarterclayton · 2015-12-02T14:49:25Z

This is a broader topic, and I agree it is important. I will add a section
to describe the impacts and summarize the current state. Unfortunately
locality is probably out of scope for rev 1 of this proposal because it
requires touch points elsewhere in the stack.

It is likely that locality is associated with volumes, not the sets - so as
long as we target the templates to allow locality to be specified it's not
a serious issue here.

If you care about locality today the DaemonSet is the appropriate tool. I
will make sure to clarify that in the design assumptions section.

On Dec 2, 2015, at 8:08 AM, Antoine Legrand notifications@github.com
wrote:

Instances can migrate from machine to machine as necessary and are not tied
to an instance

This requires to use a NetworkStorage, I would prefer to have the choice to
allow or not my Pets to be re-scheduled on an another node. If this is
possible, using local storages on the cluster would be easy.
A simple example is when someone simply wants to mount a volume like
`hostPath: /mnt/mypet-$IDENTITY'.

—
Reply to this email directly or view it on GitHub
#18016 (comment)
.

timothysc · 2015-12-02T18:10:24Z

So the key sticking point that I'm missing is gravity, forgiveness, and recovery. Once a pet has found it's home, it's not going to want to leave unless there is a maintenance / migration plan.

Otherwise many clustered systems will attempt to recover for a failure condition when in fact it was a planned outage.

smarterclayton · 2015-12-02T21:49:10Z

The first two are separate issues and while they can and should be
mentioned here, my base position is that they are orthogonal (unless we can
demonstrate otherwise).

I'm actually going to walk back on saying update is out of scope a bit - a
pragmatic requirement that reduces the need for gravity is the potential
for in place update.

Forgiveness is described in another issue but we can implement this without
forgiveness being implemented. There is not a lot of disagreement on
forgiveness that I've heard, just that how we record the policy for
forgiveness is less clear.

For recovery is this post pod death recovery? The new pod is allowed to
init whatever it wants upon startup. What examples of recovery above the
pods were you thinking of?

On Dec 2, 2015, at 1:10 PM, Timothy St. Clair notifications@github.com
wrote:

So the key sticking point that I'm missing is gravity, forgiveness, and
recovery. Once a pet has found it's home, it's not going to want to leave
unless there is a maintenance / migration plan.

Otherwise many clustered systems will attempt to recover during a failure
condition when in fact it was a planned outage.

—
Reply to this email directly or view it on GitHub
#18016 (comment)
.

bgrant0607 · 2015-12-03T20:50:47Z

docs/proposals/petset.md

+replicas exist as quickly as possible (by creating new pods as soon as old ones begin graceful deletion, for
+instance). In addition, pods by design have no stable network identity other than their assigned pod IP,
+which can change over the lifetime of a pod resource. ReplicaSets are best leveraged for shared-nothing,
+zero-coordination software.


other applicable adjectives: stateless, embarrassingly parallel, fungible

chrislovecnm · 2016-09-05T19:09:23Z

@solsson https://github.com/kubernetes/charts is probably a great place for your kafka example.

m1093782566 · 2016-09-09T03:57:13Z

/subscribe

hex108 · 2016-09-19T09:17:42Z

@smarterclayton Thanks for the proposal!

We are using PetSet, we find these addtional features are useful for our use case:

Stop and restart user specified pet, e.g. stop pet-3, then pet-3 will never be running unless user restarts it.
When rolling update, we'd like to update some pets' image version and keep testing for some time, then we could test them for a long and enough time. It means that in a long time, different pets in same PetSet use different image verion.

Would them be supported in the further? Thanks!

k8s-bot · 2016-09-19T20:32:40Z

GCE e2e build/test passed for commit b9d998f.

m1093782566 · 2016-09-20T00:11:38Z

Stop and restart user specified pet, e.g. stop pet-3, then pet-3 will never be running unless user restarts it.

We can integrate with deployment to achieve this(perhaps in the future).

When rolling update, we'd like to update some pets' image version and keep testing for some time, then we could test them for a long and enough time.

Did you try initialized annotation?

hex108 · 2016-09-20T01:35:53Z

We can integrate with deployment to achieve this(perhaps in the future).

Very glad to know it. PetSet and deployment are used for different use cases. I think PetSet needs it too. Is there any plan to support it in PetSet?

Did you try initialized annotation?

If I understand it correctly, it is used for initialization, it could not be used for rolling update after app has been running for some time. Could you explain it more? Thanks!

smarterclayton · 2016-10-20T02:15:39Z

Will update this after the rename and the pod safety proposal #34160 is reviewed.

smarterclayton · 2016-10-25T23:54:54Z

Proposal has been updated to reflect the changes to naming as decided in #27430 and has included the beta and GA criteria as described.

@bprashanth I think this is ready for merge and subsequent changes can be reflected as updates.

bprashanth

LGTM, comments were nits I'm fine with fixing them or letting it in as is

bprashanth · 2016-10-26T00:07:19Z

docs/proposals/stateful-apps.md

+* Add examples
+* Discuss failure modes for various types of clusters
+* Provide an active-active example
+* Templating proposals need to be argued through to reduce options


aren't all these already done?

bprashanth · 2016-10-26T00:08:39Z

docs/proposals/stateful-apps.md

+    rebalances.
+* Active-active
+  * Galera - has multiple active masters which must remain in sync
+* ???


master slave? (non-quorum, unilateral master)

bprashanth · 2016-10-26T00:12:30Z

docs/proposals/stateful-apps.md

+
+
+## Design Assumptions
+


in retrospect these assumptions feel a little pessimisitc. We've dicsucces:
External access direct to cluster members is out of scope
No built-in update
Limited scaling

on issues for longer than we should if we were just designing somthing that ignores any of that ;)

@bprashanth Out of curiosity - does the "limited scaling" also includes limited down-scaling? Is it possible to say I can't go lower than 3-4 replicas?

bprashanth · 2016-10-26T00:13:04Z

docs/proposals/stateful-apps.md

+## Proposed Design
+
+Add a new resource to Kubernetes to represent a set of pods that are individually distinct but each
+individual can safely be replaced-- the name **StatefulSet** (working name) is chosen to convey that the


no longer (working name)?

bprashanth · 2016-10-26T00:13:24Z

docs/proposals/stateful-apps.md

+individual can safely be replaced-- the name **StatefulSet** (working name) is chosen to convey that the
+individual members of the set are themselves "members" and thus each one is preserved. A relevant analogy
+is that a StatefulSet is composed of members, but the members are like goldfish. If you have a blue, red, and
+yellow goldfish, and the red goldfish dies, you replace it with another red goldfish and no one would


goldfish analogy doesn't work half as well

shame, it was an awesome analogy.

bprashanth · 2016-10-26T01:00:24Z

docs/proposals/stateful-apps.md

+
+Requested features:
+
+* IPs per member for clustered software like Cassandra that cache resolved DNS addresses that can be used outside the cluster (scope growth)


Might be worth noting that service per pod might also not solve this case because there is delay in updating endpoints

paralin · 2016-10-26T01:22:45Z

StatefulSets now? Oh boy. I guess this is just the nature of alpha / work in progress features. Will get started refactoring now.

... also, isn't the idea behind petsets / statefulsets keeping things healthy with persistent identities rather than culling off unhealthy pets? I certainly would NOT kill off one of my pets if it got sick. And I don't think Cassandra would appreciate random nodes dying, either.

smarterclayton · 2016-10-27T19:08:51Z

Applied prashanth's changes - labelling. Thanks for round 1 of feedback :)

bprashanth · 2016-10-27T19:11:36Z

LGTM (we're just a month away from 1 year to merge though)

smarterclayton · 2016-10-27T21:05:13Z

The amount of comments definitely breaks github, for sure.

On Thu, Oct 27, 2016 at 3:11 PM, Prashanth B notifications@github.com
wrote:

LGTM (we're just a month away from 1 year to merge though)

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
#18016 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABG_p4mkh_vp4vQ0jXQHxTvAYjPwyma2ks5q4Pd3gaJpZM4GsQyZ
.

k8s-github-robot · 2016-10-27T22:21:01Z

Automatic merge from submit-queue

k8s-github-robot · 2016-10-27T23:46:59Z

Automatic merge from submit-queue

slaskawi · 2016-12-09T11:01:27Z

docs/proposals/stateful-apps.md

+
+Requested features:
+
+* IPs per member for clustered software like Cassandra that cache resolved DNS addresses that can be used outside the cluster


Does this mean it would be possible to access a specific cluster member from the outside of the cloud? Or by outside of the cluster you mean "within the same namespace"?

Automatic merge from submit-queue Proposal for implementing nominal services AKA StatefulSets AKA The-Proposal-Formerly-Known-As-PetSets This is the draft proposal for kubernetes#260.

googlebot added the cla: yes label Dec 1, 2015

smarterclayton mentioned this pull request Dec 1, 2015

PetSet (was nominal services) #260

Closed

smarterclayton force-pushed the petset branch from fc13e71 to cc9eb23 Compare December 1, 2015 13:35

k8s-github-robot assigned brendandburns Dec 1, 2015

k8s-github-robot added kind/design Categorizes issue or PR as related to design. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Dec 1, 2015

mikedanese assigned bgrant0607 and unassigned brendandburns Dec 1, 2015

davidopp reviewed Dec 1, 2015
View reviewed changes

bgrant0607 reviewed Dec 3, 2015
View reviewed changes

solsson mentioned this pull request Oct 19, 2016

WIP three node cluster without PetSet Yolean/kubernetes-kafka#9

Closed

thockin assigned smarterclayton and unassigned thockin Oct 20, 2016

smarterclayton changed the title ~~Proposal for implementing nominal services AKA PetSets~~ Proposal for implementing nominal services AKA StatefulSets AKA The-Proposal-Formerly-Known-As-PetSets Oct 26, 2016

smarterclayton force-pushed the petset branch from b9d998f to 3f059e7 Compare October 26, 2016 00:01

bprashanth reviewed Oct 26, 2016

View reviewed changes

bprashanth added this to the v1.5 milestone Oct 26, 2016

StatefulSet proposal

14258ce

smarterclayton force-pushed the petset branch from 3f059e7 to 14258ce Compare October 27, 2016 19:08

smarterclayton added lgtm "Looks good to me", indicates that a PR is ready to be merged. release-note-none Denotes a PR that doesn't merit a release note. and removed release-note-label-needed labels Oct 27, 2016

k8s-github-robot merged commit 4773b71 into kubernetes:master Oct 27, 2016

slaskawi reviewed Dec 9, 2016

View reviewed changes

ohmystack mentioned this pull request Mar 14, 2017

StatefulSet supports all kinds of RestartPolicy #42892

Closed


		Requested features:

		* IPs per member for clustered software like Cassandra that cache resolved DNS addresses that can be used outside the cluster (scope growth)

Proposal for implementing nominal services AKA StatefulSets AKA The-Proposal-Formerly-Known-As-PetSets #18016

Proposal for implementing nominal services AKA StatefulSets AKA The-Proposal-Formerly-Known-As-PetSets #18016

Conversation

smarterclayton commented Dec 1, 2015 • edited by mikedanese

Choose a reason for hiding this comment

davidopp commented Dec 1, 2015

eswarbala commented Dec 2, 2015

ant31 commented Dec 2, 2015

ant31 commented Dec 2, 2015

ncdc commented Dec 2, 2015

smarterclayton commented Dec 2, 2015

smarterclayton commented Dec 2, 2015

timothysc commented Dec 2, 2015

smarterclayton commented Dec 2, 2015

Choose a reason for hiding this comment

chrislovecnm commented Sep 5, 2016 • edited

m1093782566 commented Sep 9, 2016

hex108 commented Sep 19, 2016

k8s-bot commented Sep 19, 2016

m1093782566 commented Sep 20, 2016 • edited

hex108 commented Sep 20, 2016

smarterclayton commented Oct 20, 2016

smarterclayton commented Oct 25, 2016

bprashanth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paralin commented Oct 26, 2016 • edited

smarterclayton commented Oct 27, 2016

bprashanth commented Oct 27, 2016

smarterclayton commented Oct 27, 2016

k8s-github-robot commented Oct 27, 2016

k8s-github-robot commented Oct 27, 2016

Choose a reason for hiding this comment

smarterclayton commented Dec 1, 2015 •

edited by mikedanese

chrislovecnm commented Sep 5, 2016 •

edited

m1093782566 commented Sep 20, 2016 •

edited

paralin commented Oct 26, 2016 •

edited