Helm chart fails when referencing licenseKey secret - pods stuck in ContainerCreating

[ServerNinja]

I'm running into an issue with the latest edge-stack helm chart. The pods get stuck in ContainerCreating due to a missing secret. This seems to happen when configuring the "licenseKey" attributes in values passed to the chart in order to reference a secret we're using for the license key we purchased.

Steps to reproduce:

1. Create namespace and a secret with license key purchased
2. Install helm chart for edge-stack - pass in values to reference secret name and key where license key is stored:

licenseKey:
  annotations: {}
  createSecret: false
  secretName: edge-stack-license
  value: license-key

Results:

• Chart deploys successfully
• Pods are stuck in ContainerCreating state

%
kubectl get pods -n edge-stack
NAME                                READY   STATUS              RESTARTS   AGE
edge-stack-58444544bb-bnfzd         0/1     ContainerCreating   0          10m
edge-stack-58444544bb-hp87z         0/1     ContainerCreating   0          10m
edge-stack-58444544bb-jnq74         0/1     ContainerCreating   0          10m
edge-stack-58444544bb-ltlqt         0/1     ContainerCreating   0          10m
edge-stack-58444544bb-pd9cp         0/1     ContainerCreating   0          10m
edge-stack-58444544bb-sktf4         0/1     ContainerCreating   0          10m
edge-stack-agent-5d5879b568-lk4k8   1/1     Running             0          10m
edge-stack-redis-56cdf99d69-zxhxk   1/1     Running             0          10m

• Events that show up on the pods:

Events:
  Type     Reason       Age                   From               Message
  ----     ------       ----                  ----               -------
  Normal   Scheduled    7m42s                 default-scheduler  Successfully assigned edge-stack/edge-stack-58444544bb-bnfzd to ip-10-64-80-135.us-west-2.compute.internal
  Warning  FailedMount  7m41s                 kubelet            MountVolume.SetUp failed for volume "edge-stack-secrets" : failed to sync secret cache: timed out waiting for the condition
  Warning  FailedMount  5m39s                 kubelet            Unable to attach or mount volumes: unmounted volumes=[edge-stack-secrets], unattached volumes=[kube-api-access-2dn86 ambassador-pod-info edge-stack-secrets]: timed out waiting for the condition
  Warning  FailedMount  89s (x10 over 7m40s)  kubelet            MountVolume.SetUp failed for volume "edge-stack-secrets" : secret "edge-stack" not found
  Warning  FailedMount  69s (x2 over 3m24s)   kubelet            Unable to attach or mount volumes: unmounted volumes=[edge-stack-secrets], unattached volumes=[ambassador-pod-info edge-stack-secrets kube-api-access-2dn86]: timed out waiting for the condition

• Secrets in namespace (the secret ("edge-stack") that the pods are trying to use for a volumeMount is missing):

%
kubectl get secrets -n edge-stack
NAME                                       TYPE                                  DATA   AGE
default-token-fpjzt                        kubernetes.io/service-account-token   3      23m
edge-stack-agent-token-284tp               kubernetes.io/service-account-token   3      10m
edge-stack-license                         Opaque                                1      23m
edge-stack-token-fswfj                     kubernetes.io/service-account-token   3      10m
sh.helm.release.v1.edge-stack-license.v1   helm.sh/release.v1                    1      23m
sh.helm.release.v1.edge-stack.v1           helm.sh/release.v1                    1      10m

Expected results:

• Chart deploys successfully
• Pods show up in a running state and secret should be present:

%
kubectl get pods -n edge-stack
NAME                                READY   STATUS    RESTARTS   AGE
edge-stack-58444544bb-29fch         1/1     Running   0          13m
edge-stack-58444544bb-5zh6f         1/1     Running   0          13m
edge-stack-58444544bb-qclf9         1/1     Running   0          13m
edge-stack-58444544bb-r98w6         1/1     Running   0          13m
edge-stack-58444544bb-tqgpz         1/1     Running   0          13m
edge-stack-58444544bb-wz6hf         1/1     Running   0          13m
edge-stack-agent-5d5879b568-xvnsf   1/1     Running   0          13m
edge-stack-redis-56cdf99d69-wl9h5   1/1     Running   0          13m

%
kubectl get secrets -n edge-stack
NAME                               TYPE                                  DATA   AGE
ambassador-internal                Opaque                                2      14m
default-token-fpjzt                kubernetes.io/service-account-token   3      65m
edge-stack                         Opaque                                1      14m
edge-stack-agent-token-9q8lf       kubernetes.io/service-account-token   3      14m
edge-stack-token-xbsrc             kubernetes.io/service-account-token   3      14m
fallback-self-signed-cert          kubernetes.io/tls                     2      14m
sh.helm.release.v1.edge-stack.v1   helm.sh/release.v1                    1      14m

Versions

• Kubernetes - EKS (AWS) 1.21
• Chart version: 7.3.2
• Edge Stack Version: 2.2.2

Other Notes

• When I remove the licenseKey values passed in, the chart installs properly and the pods show up in a running state but it doesn't use our license key.
• In our automation we're inserting the license key in a secret before the helm chart is run, which is why we need to be able to use these in the values passed into the chart.

[ServerNinja]

So I think I know the problem:

When we try to tell the helm chart which secret to reference, it's not actually using the "secretName" value.

licenseKey:
  value: license-key
  createSecret: false
  secretName: edge-stack-license

So we pass in secretName: edge-stack-license and it actually is trying to find a secret called edge-stack.

Or in other words, its looking for a secretName that is the same name as the helm release name that was passed into the helm install.

[ServerNinja]

Somehow this logic is not working:

  volumesRaw: |
    - name: {{ include "ambassador.fullname" . }}-secrets
      secret:
        {{- if and .Values.licenseKey .Values.licenseKey.secretName }}
        secretName: {{ .Values.licenseKey.secretName }}
        {{- else }}
        secretName: {{ include "ambassador.fullname" . }}
        {{- end }}

I'm passing in ".Values.licenseKey.secretName" yet it uses the "ambassador.fullname" instead.

https://github.com/datawire/edge-stack/blob/main/charts/edge-stack/values.yaml#L55

[rodmatos]

@LanceEa: any update on this? I am facing the same issue when using this Helm chart. We would like to save the API key in a Kubernetes Secret object, since there are some RBAC benefits to it.