Improvement to upgrade processes

[Firstyear]

From rc.15 to rc.16 we encountered a number of user issues due to incorrect assumptions on our part as developers.

But we learn from mistakes and improve. As a result we want to have more robust processes to limit migration issues like this.

Migrations generally fall into these categories:

• Addition of new capabilities (features)
• Loosening of restrictions to allow extra actions to be performed
• In place modification of existing data or structural elements
• Tightening of restrictions to limit what can be performed.

Generally when we have internal migrations, we aim for them to be seamless. One version upgrade we actually completely rewrote your entire database under the hood without anyone noticing. This is what we want! Generally we aim for the first 3 - addition, loosening and in place modification (such as the db example).

But sometimes we have to unfortunately tighten restrictions. Past examples (non-exhaustive) include the restriction of allowed characters in names, removing webauthn credentials that don't allow user verification and converting oauth2 configurations into "accounts". The final one was a restriction because previously oauth2 configurations were only expected to have unique names within the set of all other oauth2 configurations, but we had to convert this so that oauth2 configurations names must also be unique with respect to groups and names (For more, see #2217 )

In the last upgrade this change caused quite a few problems, and we didn't have the right tools in place to assist people to fix errors they encountered. We made a bad assumption (unlikely name collisions) when in fact it was common.

From rc.16 to 1.2.0 we have another major change where data is restricted. #2601 correctly noted that we were able to allocate posix uid/gid numbers into ranges that systemd has reserved for itself, but also that half the uid/gid range can confuse the linux kernel. As a result, we need to constrain these values.

There is a very high probability that this will affect all users with posix accounts in their instances. This is because more than 50% of the uid range we were allocating in is now not able to be used.

As a result, we want to avoid the mistakes of last release and allow you as administrators to have more time to deal with this problem. We also want to use this as a model for future migrations. A key value in our minds is that we want upgrades to be "hands free". You shouldn't have to interact with the container or server when you choose to do the upgrade.

Currently our thinking is:

• Prevent upgrades from N-1 to N if any database entries are incompatible with the new restrictions.
• backport and improve the cli tools of N-1 to support detection of non-compliant entries so that you have time to resolve the potential issues.

We are certainly open to other ideas and suggestions on how we can better handle this, especially when we have to tighten restrictions that can have significant impacts on your deployments.

[BrainWart]

As a user, I am okay with with a process where I might upgrade the tools on a test machine where the new tools may show warnings for the old database maybe. If the new tool could make suggestions or even just show the issue number related to the change, that would help me with fixing things.

[yaleman]

Chasing backports to the CLI is going to be horrible, especially across breaking changes, we're already doing this enough for it to be a burden TBH.

In that context, as we discussed,

• either kanidm or kanidmd should be able to dry-run and identify objects that need remediation before upgrades, or list any object that's going to be changed in the migration.
  • this has its own challenges, since there's potentially a lot of things being changed, in a layered way, for upgrades
• kanidmd should refuse to upgrade if there's a possibly-issue-creating change, with a list of affected objects.
• ... which kinda means the kanidmd binary should be the one doing both, for simplicity's sake (the migration filters should be the same code, for identifying objects, and also applying them)

Also we need to start rolling up the old migrations into the "base" schema to enforce the N-1, and make tracking things down over time.

[Firstyear]

Chasing backports to the CLI is going to be horrible, especially across breaking changes, we're already doing this enough for it to be a burden TBH.

In that context, as we discussed,

* either `kanidm` or `kanidmd` should be able to dry-run and identify objects that need remediation before upgrades, or list _any_ object that's going to be changed in the migration.
  
  * this has its own challenges, since there's potentially a lot of things being changed, in a layered way, for upgrades

* `kanidmd` should refuse to upgrade if there's a possibly-issue-creating change, with a list of affected objects.

* ... which kinda means the `kanidmd` binary should be the one doing both, for simplicity's sake (the migration filters should be the same code, for identifying objects, and also applying them)

But then we are back to "admin has to have a hands on upgrade". I'd rather a way we can pre-warn about changes and when they are ready they can opt in and even better if the cli tools the admin uses for day to day admin is the touch point there rather than having to get access to the admin unixd socket.

I was thinking about this and the domain migration framework does allow this. Internally the server can read the current domain level and then use that for logic decisions.

This means rather than backporting the detection tools, we commit them for the next release.

The user upgrades (DL5 -> 6). This does all the "painless" stuff. The new cli tools then can warn the users "okay, here are the upcoming issues with uid/gid numbers. Then the user can manually request 6 -> 7.

Then the next release we ship 7 -> 8 as automatic and 8 -> 9 as opt in. This way if a user goes from 1.2.0 -> 1.3.0 but never did the upgrade manually, we enforce it on the major version update and then they have the tools at hand to fix the issue etc.

internally to the server we can read "if dl < 7" and take one validation path and vice versa.

it makes it a little more complex internally, but I think it gives a better outcome, and it means we get better feature gating.

It also means that in a replicated environment, the upgrade happens on all nodes at the same time effectively too.

In a way, the above proposal is just recreating my original one, but rather than backport, we just give more time after the release to fix things. It also avoids the potential trap of users on the previous version that didnt upgrade their CLI from missing out on an important upgrade warning.

The other reason this approach is good is we now have the upgrade test frame work so we can test this enforcement over versions of the server internally, but also it lays some ground work to mixed server version upgrades with replication in a future version.

Also we need to start rolling up the old migrations into the "base" schema to enforce the N-1, and make tracking things down over time.

Yeah, this will be part of migration cleanups once we start to jettison some older versions of the server from upgrade paths.

[philipcristiano]

Are “more releases” an option that would help? Instead of back porting changes plan 2 releases (R1, R2) where the first warns on identified problems and then (maybe optionally) enforces not-causing problems if the system is “ready” for the second release. The second release then enforces everything required. Release notes for R2 mention needing to move from R1 for an in place, hands off upgrade.

[yaleman]

Are “more releases” an option that would help?

I sure hope not, because it requires aligning and releasing about ... 15 crates now? 😱

[philipcristiano]

The release process (assuming this) does seem quite involved. There is tooling that can help automate a bunch of it though. Are you open to automating the release process?

[Firstyear]

The release process isn't the problem here. Even if we released all the time to resolve issues, we still have to think about mixed version replication environments in future, and ways to help admins resolve issues in their data and systems. We still need ways to release tools that assist admins to upgrade smoothly. Releasing more doesn't fix that.

[tumbl3w33d]

You're already doing great. I'll just throw some crazy ideas in here and you'll know best.

1. You could create an in-memory duplicate of the entire db, run migrations and only on-success say "yey, you're good to go"
2. You could let us start a second container with the new version next to the one with the old version (or simply add a canary to the replication cluster) and make them talk to each other, either via special style of replication (on-the fly migration) or SCIM. We would then shutdown the old one and continue with the new one.

Both ways, you would inform the operator about roadblocks during the migration.

[Firstyear]

1. We actually already do this during the upgrade so if it fails, then it implicitly rolls back. During this we will log entries that are not compliant in the future, but we still want a better set of warning tools ahead of time.
2. That would required mixed version replication which is a longer term thing. But it's possible.