Ability to upgrade separate components of the cluster

[dnavre]

Upgrading to Kubernetes 1.5 in case one has PetSets is a quite complex, it requires you to separately upgrade the master node and then after altering the server config upgrade the nodes running kubelet. As I know current kops cluster upgrade mechanism doesn't allow this kind of manipulations.

Is this OK cause PetSets/StatefulSets are alpha/beta features or should kops support this?
I personally would prefer Kubernetes core to upgrade PetSets to Stateful sets automatically during startup.

P.S.
My first 5 cents into the Kubernetes project. You've done some great works guys!

[chrislovecnm]

@dnavre ping me if I do not get back to you on this, but this is an interesting thing. I don't think we will support it inside of kops, but I think up building an upgrade tool makes sense.

cc: @kubernetes/sig-apps-misc Any folks planning on a Petset -> Stateful Set upgrade tool?

[enisoc]

@chrislovecnm When you say kops won't support "it", do you mean kops won't support pausing between master upgrade and node upgrades to let the user perform manual upgrade steps? Or do you just mean kops won't upgrade API objects for you automatically? If the former is not supported, then we made a mistake in assuming our manual upgrade flow is a viable workaround across all platforms.

@dnavre Regarding having core Kubernetes upgrade things automatically, I believe that's the path used for changes to GA APIs (and maybe Beta?). Unfortunately, it requires a fair bit of plumbing to support and it was decided that the extra work was not warranted in this case given that PetSet was Alpha: kubernetes/kubernetes#27430 (comment).

[dnavre]

@enisoc Thanks for the response. I agree that it's absolutely ok to have manual upgrade procedures for alpha features. However in this case, as I understand it, a cluster built using kops is practically un-upgradable because of the pause needed between master and node upgrades. One will need to manually backup all the PVCs, delete the nodes and reapply all that on the upgraded cluster.
Also, I think that in the future this kind of behavior(upgrading instance groups one-by-one) may come to be very useful to some users.

[justinsb]

I agree that it would be useful to support one instancegroup at a time upgrades. I thought we had an issue for it, but I just opened #1265 (we can close if it turns out to be a duplicate).

I think we've now figured out how to do renames in a more hands-off manner (with ScheduledJobs -> CronJobs) so hopefully this is a one-off problem.

[justinsb]

#1270 should add rolling-update on a per-instancegroup level. I agree it is not ideal for this case, but it sounds like this unblock people. And if k8s is not going to do an automated update, I don't know if there's anything more we can do in kops (other than lobby for no more renames without migration support...)

[justinsb]

As I believe this is fixed and beta1 is available, I'm going to close. But if you hit it again with kops 1.5.0 beta1 or later, please do comment/reopen.