Helm chart for Kamus does not support multiple installations in one cluster without --force

[kartsm]

Please update the chart so that --skip-crds works, and that multiple installs are possible within the cluster without --force.

We had three Kubernetes clusters, and we are in the process of consolidating down to two. (They are actually OpenShift but I don't think that's the key difference here.) Each of the older clusters had an installation of Kamus, which we did not expose outside the cluster. In order to make migration as smooth as possible for existing workloads, we have rearranged namespaces from the three clusters; some have gone on to one cluster, and some the other. So necessarily some of them must be reconfigured to use a different installation of Kamus, and multiple installations appear to be the easiest way to get there.

My environment:

$ uname -a
Darwin CORP-L-9S9MD6M 20.6.0 Darwin Kernel Version 20.6.0: Wed Jan 12 22:22:42 PST 2022; root:xnu-7195.141.19~2/RELEASE_X86_64 x86_64

$ helm version
version.BuildInfo{Version:"v3.8.0", GitCommit:"d14138609b01886f544b2025f5000351c9eb092e", GitTreeState:"clean",GoVersion:"go1.17.6"}

I am using version 0.9.9 of the chart, from the repository on https://charts.soluto.io.

$ helm repo list
NAME            URL
soluto          https://charts.soluto.io

$ helm show chart soluto/kamus
apiVersion: v1
appVersion: 0.9.0.7
description: An open source, git-ops, zero-trust secrets encryption and decryption
  solution for Kubernetes applications
home: https://kamus.soluto.io
icon: https://raw.githubusercontent.com/Soluto/kamus/master/images/logo.png
keywords:
- gitops
- secrets
maintainers:
- name: omerlh
- name: shaikatz
name: kamus
sources:
- https://github.com/Soluto/Kamus
version: 0.9.9

The first deployment to a cluster works as expected.

$ date; time helm upgrade kamus-dev soluto/kamus --install --atomic --namespace mars-coyote-nest --values clusters/infra-lab/helm/kamus/dev.yaml  ; date
Wed Mar  9 09:33:55 PST 2022
Release "kamus-dev" does not exist. Installing it now.
NAME: kamus-dev
LAST DEPLOYED: Wed Mar  9 09:34:01 2022
NAMESPACE: mars-coyote-nest
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace mars-coyote-nest -l "app=kamus,release=kamus-dev,component=encryptor" -o jsonpath="{.items[0].metadata.name}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl port-forward $POD_NAME 8080:9999

real    0m45.627s
user    0m1.331s
sys     0m1.026s
Wed Mar  9 09:34:41 PST 2022

The second deployment to a cluster, in a separate namespace, complains about ownership of the CustomResourceDefinition, even when --skip-crds is specified.

$ date; time helm upgrade kamus-dev soluto/kamus --install --atomic --namespace nasty-comet-sore --values clusters/infra-lab/helm/kamus/stage.yaml --skip-crds ; date
Wed Mar  9 09:36:01 PST 2022
Release "kamus-dev" does not exist. Installing it now.
Error: rendered manifests contain a resource that already exists. Unable to continue with install: CustomResourceDefinition "kamussecrets.soluto.com" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-namespace" must equal "nasty-comet-sore": current value is "mars-coyote-nest"

real    0m7.279s
user    0m1.013s
sys     0m0.746s
Wed Mar  9 09:36:08 PST 2022

In The Chart File Structure Helm recommends putting CRDs in the crds/ directory, sibling to templates/. So I clone the repository containing the chart and made that change locally against the kamus-0.9.9 tag.

$ pwd
/REDACTED/src/soluto-helm-charts

$ git remote -v
origin  https://github.com/Soluto/helm-charts.git (fetch)
origin  https://github.com/Soluto/helm-charts.git (push)
    
$ git log -n 1
commit 44067cb095a7264d6c335f9f6d2e78f22d1e1d53 (HEAD, tag: kamus-0.9.9, origin/master, origin/HEAD, master)
Merge: 69a445a 64f57a3
Author: Shai Katz <shaikatz@users.noreply.github.com>
Date:   Thu Mar 18 11:47:13 2021 +0200
    
    Merge pull request #55 from Soluto/add-reconciliation-interval-to-values

$ git status
HEAD detached at kamus-0.9.9
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        renamed:    templates/kamussecret-crd.yaml -> crds/kamussecret-crd.yaml

I deleted the old releases and tried again:

$ helm delete -n nasty-comet-store kamus-dev
Error: uninstall: Release not loaded: kamus-dev: release: not found
    
$ helm delete -n mars-coyote-nest kamus-dev
release "kamus-dev" uninstalled
    
$ date; time helm upgrade kamus-dev ../soluto-helm-charts/charts/kamus --install --atomic --namespace mars-coyote-nest --values clusters/infra-lab/helm/kamus/dev.yaml  ; date
Wed Mar  9 10:33:10 PST 2022
Release "kamus-dev" does not exist. Installing it now.
NAME: kamus-dev
LAST DEPLOYED: Wed Mar  9 10:33:16 2022
NAMESPACE: mars-coyote-nest
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace mars-coyote-nest -l "app=kamus,release=kamus-dev,component=encryptor" -o jsonpath="{.items[0].metadata.name}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl port-forward $POD_NAME 8080:9999

real    0m45.806s
user    0m1.208s
sys     0m0.986s
Wed Mar  9 10:33:56 PST 2022
    
$ date; time helm upgrade kamus-dev ../soluto-helm-charts/charts/kamus --install --atomic --namespace nasty-comet-sore --values clusters/infra-lab/helm/kamus/dev.yaml  --skip-crds ; date
Wed Mar  9 10:34:25 PST 2022
Release "kamus-dev" does not exist. Installing it now.
Error: rendered manifests contain a resource that already exists. Unable to continue with install: ClusterRole "kamus-controller" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-namespace" must equal "nasty-comet-sore": current value is "mars-coyote-nest"

real    0m6.574s
user    0m0.844s
sys     0m0.668s
Wed Mar  9 10:34:32 PST 2022

Now the complaint is about the ClusterRole kamus-controller. The deployment does work if I force it.

$ date; time helm upgrade kamus-dev ../soluto-helm-charts/charts/kamus --install --atomic --namespace mars-coyote-nest --values clusters/infra-lab/helm/kamus/dev.yaml  --force ; date
Wed Mar  9 10:40:23 PST 2022
Release "kamus-dev" has been upgraded. Happy Helming!
NAME: kamus-dev
LAST DEPLOYED: Wed Mar  9 10:40:29 2022
NAMESPACE: mars-coyote-nest
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
  export POD_NAME=$(kubectl get pods --namespace mars-coyote-nest -l "app=kamus,release=kamus-dev,component=encryptor" -o jsonpath="{.items[0].metadata.name}")
  echo "Visit http://127.0.0.1:8080 to use your application"
  kubectl port-forward $POD_NAME 8080:9999

real    0m10.452s
user    0m1.575s
sys     0m1.001s
Wed Mar  9 10:40:34 PST 2022

My current workaround is to use --force, and we will eventually be changing how our workloads use Kamus. Someone else in the future may be in a situation similar to mine and may be more hesitant to using the blunt instruments that are --skip-crds and --force.