/kind bug

What steps did you take and what happened:

Up to now, updating the Cluster Add-ons has occurred concurrently with the upgrade of the cluster.

See clusteraddon_controller.go L182-199.

This leaves the outcome to chance. All three outcomes are possible:

• An Add-on is upgraded before a new Kubernetes version is rolled out.
• An Add-on is upgraded after a new Kubernetes version is rolled out.
• An Add-on is upgraded at the same time as the new Kubernetes version.

Case 3 is especially dangerous.

If a fundamental Add-on like the Container-Network-Interface is upgraded during the rollout of a new Kubernetes Version, the reachability between the nodes could be jeopardized.

What did you expect to happen:

As a creator of Cluster-Stacks, I want to configure whether an add-on is upgraded before or after the upgrade of a new Kubernetes version.

As a creator of Cluster-Stacks, I want to configure that add-on X is upgraded (or deleted) after add-on Y.

As a creator of Cluster-Stacks, I want to be able to add additional steps before or after an upgrade. For example, I want to automatically create a backup before the upgrade happens.

Example

The CNI Cilium requires that a pre-flight check be executed before the new Cilium version is applied.

Currently, it is not possible to automate these steps:

1. Apply the pre-flight check.
2. Wait until the pre-flight check signals "OK to upgrade."
3. Apply the upgrade.
4. Remove the pre-flight check.

Up to now, these steps have required manual work.

Cluster API Lifecycle Hooks

The Cluster API Lifecycle Hooks could be used to implement these steps.