Upgrade flow for replicated setups

[tumbl3w33d]

I'm about to introduce replication to my production setup and played with it on a staging environment before that. While all seems to work fine, I was wondering how I would execute future upgrades. The docs currently don't mention that part, unless I missed something.

I have the primary-secondary setup (active-passive) as described in the docs, so my primary node is always right in case of conflicts and the secondary only takes over if the primary goes down. In front of it I have a HAProxy managing the failover.

My first thought was, I could just nuke my secondary and act like there's no replication to keep things simple for the upgrade. However, the/one point of replication is to have higher availability so while this works (and might be sufficient for some who just want to schedule a maintenance window) it's not as nice as a rolling upgrade.

Now let's assume I want to be as available as possible. What would be the recommended flow/order to upgrade the cluster? I could image it's something like:

• take primary node out of load balancer
• secondary node now operates like it's the fully operational primary (it stores changes from writes and serves data for reads)
• the primary node can now be upgraded by replacing the running container with the newer and possibly execute all necessary other migration steps
• when I start the primary again (it's not active in the load balancer yet), replication would probably start again, which means the primary would retrieve changes that happened on the secondary in the meantime
  • is there a potential for trouble at this point, because the data structures etc. possibly changed on the primary?
• assuming that the instances are in-sync again (should I verify that first by asking some api endpoint?) I would add the primary node to the load balancer again
• now it's time to take the secondary out of load balancer and execute the upgrade there, then return it

Looking forward to get some insights. :)

[yaleman]

At the moment they straight up just refuse to replicate between disparate versions, so that's fairly easy. @Firstyear was talking about a future path however with out-of-sync-version-upgrades, which is partially helped by the upgrade check framework we're currently putting in place.

[Firstyear]

• take primary node out of load balancer

• secondary node now operates like it's the fully operational primary (it stores changes from writes and serves data for reads)

• the primary node can now be upgraded by replacing the running container with the newer and possibly execute all necessary other migration steps

• when I start the primary again (it's not active in the load balancer yet), replication would probably start again, which means the primary would retrieve changes that happened on the secondary in the meantime

  • is there a potential for trouble at this point, because the data structures etc. possibly changed on the primary?

• assuming that the instances are in-sync again (should I verify that first by asking some api endpoint?) I would add the primary node to the load balancer again

• now it's time to take the secondary out of load balancer and execute the upgrade there, then return it

These steps are all fine. The big point here is that currently once you restart the primary, it will refuse to sync with the secondary until you also upgrade the secondary. Then they will re-sync and continue. As @yaleman said, we require "identical versions" on all nodes for the moment.

Replication will automatically tell you if things get out of sync, Kanidm has internal checks for this.

In the future the process would be similar, but version N could sync with N minus 1, but would only operate at a functional level (aka domain level) of N minus 1. Once the upgrades are complete, you would signal the server to raise the domain level, which then would "replicate out" to all nodes that they can raise their behaviour to version N.

This way you can do a roll out over multiple nodes, then once complete, flag that the newer version features can be used. This way an older node wouldn't get changes it can't understand.

However, to achieve that requires a lot more testing around this, and while we have some of that in place now, and while all the frameworks and tools are there, we don't want to support this yet because it adds extra risks for a still "relatively young" feature.

[tumbl3w33d]

Just to make sure this case is handled: I upgrade the inactive primary and during that time the secondary receives write operations. Now the secondary goes inactive for upgrade and the primary takes over. I understand that in this time frame the primary will lag behind until secondary upgrade is finished, but if the primary also receives a write, will they be able to handle the situation once they are both upgraded and can talk to each other again? The secondary is told to auto refresh, which could mean in conflict case the later write on the primary would win, right? Sounds like I should just take both offline for the moment and wait for future refinement.

[yaleman]

My understanding is that that attribute-based eventual-consistency we use would effectively merge the state once they agree to talk to each other again.

[Firstyear]

Yep that's correct. The attribute changes would be merged post upgrade. The merge order is "last write wins", with some exceptions for special attributes. This is extensively tested :)

In other words, @tumbl3w33d don't worry. This will work correctly. When you take the primary offline, the secondary receives a write, and then the primary comes back replication pauses (due to different versions). When the secondary is upgraded and reconnects, it will send your changes to the primary and they merge their state.