Use operator to update PeerPod components

[huoqifeng]

I'm creating this issue to track and discuss the scenario to upgrade cloud-api-adaptor components via operator after PeerPod enabled. for example:

• New PeerPod image available to fix security problems and to update the cloud-api-adpators configuration only
• Update the cloud-api-adpators configuration and the cloud-api-adpator binaries
• Update the kata-runtime binaries and configuration -- CoCo global CcRuntime operator handle it
• Upgrade containerd -- Because we're going to enable image offload via annotation, expecting is not to use a forked containerd in operator later, so no upgrade for containerd.
• Other components upgrade???
• When updating the deamonset for cloud-api-adaptor, it should support rolling update.

Approach?

• CoCo operator support kata runtime update by revise the CcRuntime CR
  • support rolling update already
• PeerPod daemonset configmap support dynamic update
  • sounds not automatic Facilitate ConfigMap rollouts / management kubernetes/kubernetes#22368, alternative https://github.com/stakater/Reloader/
  • Change cloud-api-adaptor code to watch the configmap mount file changes?
  • add dummy code in daemonset object?
• PeerPod daemonset support rolling update
• Shall we add caa operator to handle PeerPod daemonset update case?

[huoqifeng]

@stevenhorsman @bpradipt @wainersm @jensfr @jtumber-ibm @liudalibj @fidencio for your awareness.

[wainersm]

Hi @huoqifeng ,

I'm creating this issue to track and discuss the scenario to upgrade cloud-api-adaptor components via operator after PeerPod enabled. for example:

* New PeerPod image available to fix security problems and to update the cloud-api-adpators configuration only

* Update the cloud-api-adpators configuration `and` the cloud-api-adpator binaries

* ~Update the kata-runtime binaries and configuration~ -- CoCo global CcRuntime operator handle it

* ~Upgrade containerd~ -- Because we're going to enable image offload via annotation, expecting is not to use a forked containerd in operator later, so no upgrade for containerd.

* Other components upgrade???

Do you consider to update the webhook and peerpods-ctr too?

[snip]

[huoqifeng]

@wainersm yes, I'd like webhook and peerpods-ctr can be updated also, Looks it's not added in operator yet and several stories might be related?
#1103
#872
#864
@stevenhorsman @jtumber-ibm @caavery

[jensfr]

@wainersm yes, I'd like webhook and peerpods-ctr can be updated also, Looks it's not added in operator yet and several stories might be related? #1103 #872 #864 @stevenhorsman @jtumber-ibm @caavery

Yes, all those stories are related to the effort of having the coco operator deploy peer pod components. It is targeted for v0.8.0

[jensfr]

First of all, thank you for bringing up this issue, @huoqifeng. We need to clarify two different scenarios here.

Scenario One involves deploying an operator via the Operator Lifecycle Manager (OLM). In this case, an update is achieved by modifying the manifest files and constructing a new bundle. This new bundle is subsequently added to a catalog. You can refer to the following documentation for a more comprehensive understanding: https://sdk.operatorframework.io/docs/olm-integration/tutorial-bundle/

The Coco operator was developed using operator-sdk, and we generate a new bundle with each release, which is then uploaded to operatorhub.io. Once the issues referenced above are resolved, components like Peer pods(-config) controller will be included in this bundle.

However, I understand that not all users implement OLM in their environments, leading us to Scenario Two. In such a scenario, to upgrade, we as a project could offer a version-specific YAML file encompassing all the manifests that constitute coco operator, peer pod (config) controller, caa daemonset, secret, configmap, rbac, webhook configuration etc. For example, to upgrade from version 0.7.0, a user would execute 'kubectl apply -f releases/v0.8.0.yaml', which would refresh the deployments and daemon sets. An example for this approach is the gitlab operator: https://docs.gitlab.com/operator/operator_upgrades.html
Alternatively, we could look into kustomize to patch each deployment, daemonset, configmap, and so forth.

For some components, we may need to tweak the peer-pod(-config) controller to monitor changes to specific CRDs and handle them appropriately.

In both scenarios, we might consider leveraging https://github.com/stakater/Reloader/ if it could assist us in streamlining the additional controller code.

[bpradipt]

Today, peerpodconfig-ctr watches for changes to only peerpodconfig crd. Should we also look at extending it to watch for the configmap and secrets used by CAA and take suitable action as needed?

[huoqifeng]

@jensfr thanks for the clarification, regard the Scenario 2, I suppose it's our IKS case. if we want update the caa daemonset itself or corresponding secret and configmap, how should we do? I'm supposing end user can revise a CR object which the peerpodconfig-ctr watches but not change the daemonset/secret/configmap directly? @bpradipt