k3s: document upkeep process

[superherointj]

k3s: document upkeep process

The user guidelines has to be improved too. But this is a start.

Let me know of corrections, improvements and objections.

CC K3s maintainers: @euank @yajo @Mic92

[yajo]

Could you please not mention my user in the commit msg? Github spams me otherwise.

You can use:

Co-authored-by: <973709+yajo@users.noreply.github.com>

[superherointj]

How should backport to stable work?
Currently it is broken. But after next release, it will be in sync again. So back-porting might be important.
I don't use stable myself. I continuously pin unstable. (Make unstable stable.) So I don't get to experience the stable issues.

[superherointj]

@euank I need your feedback here. Release is happening. And I'm not sure I have correctly documented the process.
It seems I wrongly understood the removal of K3s versions happening after release. And not before release. What should I do?

[yajo]

It seems to me like the process you outlined here is very good 🤔

[euank]

Thanks for writing this up @superherointj, appreciated!
Apologies for not keeping up with nixpkgs stuff for this release, your help on that is also very much appreciated!

You're right that the intention originally was to remove k3s versions from the next version's release branch, but to leave all supported ones on unstable.

I do feel like deleting all but the newest version on unstable periodically would be really convenient for those of us who only use unstable, but I think we're stuck with following something like I wrote before if we want to allow people to follow k8s's upstream upgrade recommendations, and also stick to nixos stable releases.

I think leaving versions in unstable is also required for backporting patches to older versions while they're supported, i.e. so we can backport 1.28 patches to 23.11 for the month-or-so after 24.05 when it's still supported. I think that's also necessary if we want people to be able to follow upstream k8s's upgrade recommendations (1 version at a time, always using the latest patch version iiuc)

Based on the timeline in #303285, it does sound like we have time to sort this out though hopefully!

If we're following what I wrote before, the required next actions would be:

• Backport k3s 1.28 to release-23.11 (so people can upgrade safely from release-23.11 to 1.29 in release-24.05)
• As soon as release-24.05 branches off, delete all but k3s 1.29 from it
• Update the maintainer document a bit more to make this all clearer

Again, apologies for being busy and not putting as much attention into this as I should!

I can open a PR updating some of this documentation in the next couple days as a means to discuss it further, if that sounds good; I'm also happy to change my mind on any of this if we feel like taking some other approach is more pragmatic. The "trimming versions after we branch off" really is a tight timing window, and seems fraught, so I'd love an alternative to that 😅

What do y'all think?
Do the above steps make sense for this release still, or do we have a better option to pursue?

[superherointj]

@euank Thanks for correcting me. As always, glad to have you here!

I've reached a new understanding:

As contributors are limited. To be realistic. The criteria should not rely on individual contributors doing timely work.

Basically:

• The pace of removal should match the pace of back-porting.
• After release, all versions from unstable that exist in stable should be removed.

Currently, for example, would mean:

• k3s_1_27 would be removed because it exists in stable.
• k3s_1_28 would be kept because it doesn't exist in stable. Unless, k3s_1_28 gets ported to stable, then, it should be removed.

This way special upkeep for stable is optional (desirable, but not mandatory).
From unstable, back-porting should be triggered as usual. On failure, the maintainer gets to decide what to do.

Back-porting to stable speeds up package removal. With increased back-porting contributions, version removal will speed up naturally. It seems, this criteria naturally works for all situations.

I'm against forcing an external artificial deadline for packages, while not having the manpower to keep up with such level of contributions. As the package gets used more, new contributors may contribute and then the quality/speed of upkeep will improve naturally. We can't force better output than this. I honestly cannot keep up with much more than what I do already. If some package goes EOL, we have to be realistic. It is just debt that doesn't get solved by a deadline. Polar thinking is bad.

On agreement, I may open a PR fixing documentation with this new understanding.

[euank]

I agree reducing maintainership load for k3s is important, and minimizing the number of k3s versions we support is good!

I also think what you've proposed means there's always a supported k3s upgrade path, though it does force people on unstable to upgrade more aggressively.

I think the main reason we ended up where we are in terms of versions was based on discussion in this issue: #224483

@yajo, since you had some thoughts then, just to double check, the above proposal works for you right?

I'm okay with the proposal personally, I just use unstable (and can pluck versions off release branches if I need to).
It doesn't feel ideal if we had unlimited manpower, but we don't, so it does seem like it might be a good compromise.

Thanks for writing it up @superherointj, happy to review a docs PR and help push towards that setup if @yajo's also onboard!

[superherointj]

it does force people on unstable to upgrade more aggressively.

To avoid coupling system upgrade and K3s upgrade, it would be necessary to keep last stable version in unstable. This would avoid breaking K3s instantly on system upgrades, and the user could migrate K3s version at his own timing, in a discrete manner. As Kubernetes has differences in application, such as manifests version, CRDs, adds-ons having to catch up to Kubernetes version. This makes sense. So, I agree, that last proposal was too aggressive to unstable (could affect applications not ready for lastest version) and stable (coupled system upgrade). The proposal of a hasty upgrade was in trying to comply with previous nixpkgs release manager demands.

So the compromise of the compromise, would be to keep at least one/the last version of stable in unstable. Then, stable users can upgrade NixOS without going into an instant K3s upgrade crisis.

Thoughts?

[euank]

I'm happy with that proposal, yup!

To be sure I understand the proposal, another way to say it is:

1. We try to keep the latest version of k3s up to date on unstable
2. We try to backport new versions of k3s to the stable release
3. If the newest version on the stable release is different from unstable, we also keep that "latest stable version" on unstable.

This leads to unstable having either 1 or 2 k3s versions, and stable having all versions between the version on unstable when it was cut, up to the current version on unstable, or one version before.

Specifically, that might lead to things like (using hypothetical future versions) "Stable has 1.28 (version when it was cut) + 1.29 + 1.30, unstable has just 1.30, or unstable has 1.30 + 1.31".

Does that sound accurate?

If I'm summarizing what you're saying correctly, that sounds good to me!

[superherointj]

TLDR: Skip this.

---

Thinking from unstable reference (in opposition to stable reference):

• All changes are done to unstable. stable is a past unstable.

• The K3s package removal happens only at unstable.

• unstable looks to stable for knowing the versions that can be removed.

• stable inherits removals from unstable at release.

• Back-porting enables version removal from unstable at next release.

• Back-porting to stable is possible but optional, not guaranteed.

  • stable receives back-ports whenever possible.
    • All that merge fine for sure.
    • Manual work for back-porting would be optional and executed by interested contributors.

• The stable set of versions may lag in reference to unstable. This does not block updates, only delays version removal from unstable.

• For a smooth system (not K3s) upgrade, unstable only requires latest-stable package version.

• The version set stable has is not important for the upgrade process.

• unstable keeps latest-stable version + newer versions (that stable doesn't have).

• stable keeps unstable release versions + (optional) unstable backports.

• Whenever latest-stable version equates latest-unstable version, unstable will have a single K3s package version.
  But lagged upkeep requires 2+ versions. See:
  Considering latest-unstable as 1.31:

  • If latest-stable is 1.31, unstable keeps 1.31 only.
  • If latest-stable is 1.30, unstable keeps 1.30+1.31.
  • If latest-stable is 1.29, unstable keeps 1.29+1.30+1.31.
  • If latest-stable is 1.28, unstable keeps 1.28+1.29+1.30+1.31.

I'll prepare the documentation. Then, we can iterate in the PR.

Other considerations:

• We should recommend users to continuously upgrade K3s. (To be at last available K3s version.)

[yajo]

It's becoming a bit complex to understand for me, I guess It'll be easier on the PR.

However, to put it simple, unstable and stable should share at least one minor k3s version.

I a gree with this sentence:

To avoid coupling system upgrade and K3s upgrade, it would be necessary to keep last stable version in unstable

Reasoning:

• A system upgrade is hard enough.
• If something breaks, it'll be easier to know if it's k3s itself or the rest of your NixOS.

[superherointj]

Sorry about my previous convoluted answer.

• After every nixos release, K3s maintainers should remove from nixos-unstable all K3s versions that exist in nixos-stable but the latest version (which is necessary to avoid coupling system upgrade to k3s upgrade).

Does this sums up?