argocd-vault-plugin (sidecar) not updating secret value

[razahuss02]

I installed argocd-vault-plugin via sidecar and I am having trouble and getting a test secret in my cluster to update with the credentials that live in vault. The AVP container is coming up and it seems that things are healthy and running.

Vault was installed as HA with TLS enabled.

I installed argocd via Helm, and the argocd-apps via helm as well.

This is what I have for my repoServer:

  repoServer:
    autoscaling:
      enabled: true
      minReplicas: 3
      maxReplicas: 6

    env: 
      - name: AVP_AUTH_TYPE
        value: k8s
      - name: AVP_K8S_ROLE
        value: argocd
      - name: AVP_TYPE
        value: vault
      - name: VAULT_ADDR
        value: "https://vault.vault:8200"
      - name: VAULT_SKIP_VERIFY
        value: "true"

    # Download tools
    initContainers: 
      - name: download-tools
        image: registry.access.redhat.com/ubi8
        env:
          - name: AVP_VERSION
            value: 1.16.1
        command: [sh, -c]
        args:
          - >-
            curl -L https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v$(AVP_VERSION)/argocd-vault-plugin_$(AVP_VERSION)_linux_amd64 -o argocd-vault-plugin &&
            chmod +x argocd-vault-plugin &&
            mv argocd-vault-plugin /custom-tools/
        volumeMounts:
          - mountPath: /custom-tools
            name: custom-tools
    
    extraContainers:
      - name: avp
        command: [/var/run/argocd/argocd-cmp-server]
        image: quay.io/argoproj/argocd:v2.9.3
        securityContext:
          runAsNonRoot: true
          runAsUser: 999
        volumeMounts:
          - mountPath: /var/run/argocd
            name: var-files
          - mountPath: /home/argocd/cmp-server/plugins
            name: plugins
          - mountPath: /tmp
            name: tmp
          # Register plugins into sidecar
          - mountPath: /home/argocd/cmp-server/config/plugin.yaml
            subPath: avp-helm.yaml
            name: cmp-plugin
          # Important: Mount tools into $PATH
          - mountPath: /usr/local/bin/argocd-vault-plugin
            subPath: argocd-vault-plugin
            name: custom-tools

    volumes: 
      - configMap:
          name: argocd-cmp-cm
        name: cmp-plugin
      - name: custom-tools
        emptyDir: {}

    
    resources:
      limits:
        cpu: 200m
        memory: 512Mi
      requests:
        cpu: 100m
        memory: 256Mi

    topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          app.kubernetes.io/name: argocd-repo-server

    rbac:
      - verbs:
          - get
          - list
          - watch
        apiGroups:
          - ''
        resources:
          - secrets
          - configmaps   

Here is my cmp definition

    cmp:
      create: true
      plugins: 
        avp-helm:
          init:
            command: 
              - sh
              - "-c"
              - |
                helm repo add argo https://argoproj.github.io/argo-helm
                helm repo add bitnami https://charts.bitnami.com/bitnami
                helm dependency build
          allowConcurrency: true
          discover:
            find:
              command:
                - sh
                - "-c"
                - "find . -name 'Chart.yaml' && find . -name 'values.yaml'"
          generate:
            command:
              - sh
              - "-c"
              - |
                helm template $ARGOCD_APP_NAME --include-crds -n $ARGOCD_APP_NAMESPACE -f <(echo "$ARGOCD_ENV_HELM_VALUES") . |
                argocd-vault-plugin generate . -s argocd-vault-plugin-credentials -
          lockRepo: false

Here is the application CRs for argocd

argocd-apps:
  applications:
    - name: argocd
      namespace: argocd
      finalizers:
        - resources-finalizer.argocd.argoproj.io
      project: cluster-addons
      source:
        helm:
          valueFiles:
            - values.yaml
        repoURL: **********
        targetRevision: branch
        path: path/to/values/file
      destination:
        server: https://kubernetes.default.svc
        namespace: argocd
      syncPolicy:
        automated:
          prune: false
          selfHeal: true
          allowEmpty: true
      ignoreDifferences:
        - group: apps
          kind: Secret
          jsonPointers:
          - /data/admin.passwordMtime

secret container vault configuration for argo

kind: Secret
apiVersion: v1
metadata:
  name: argocd-vault-plugin-credentials
type: Opaque
stringData:
  AVP_AUTH_TYPE: "k8s"
  AVP_K8S_ROLE: "argocd"
  AVP_TYPE: "vault"
  VAULT_ADDR: "https://vault.vault:8200"
  VAULT_SKIP_VERIFY: "true"

Test secret

apiVersion: v1
kind: Secret
metadata:
  name: test-secret
type: Opaque
stringData:
  username: "<path:secrets/data/test#username>"

When I run argocd-vault-plugin generate . -s argocd-vault-plugin-credentials within the avp container, this is what I get returned:

argocd@argocd-repo-server-75f6898758-nqlsp:~$ argocd-vault-plugin generate -s argocd-vault-plugin-credentials .
apiVersion: argoproj.io/v1alpha1
kind: ConfigManagementPlugin
metadata:
  name: avp-helm
spec:
  allowConcurrency: true
  discover:
    find:
      command:
      - sh
      - -c
      - find . -name 'Chart.yaml' && find . -name 'values.yaml'
  generate:
    command:
    - sh
    - -c
    - |
      helm template $ARGOCD_APP_NAME --include-crds -n $ARGOCD_APP_NAMESPACE -f <(echo "$ARGOCD_ENV_HELM_VALUES") . |
      argocd-vault-plugin generate . -s argocd-vault-plugin-credentials -
  init:
    command:
    - sh
    - -c
    - |
      helm repo add argo https://argoproj.github.io/argo-helm
      helm repo add bitnami https://charts.bitnami.com/bitnami
      helm dependency build
  lockRepo: false
---

I guess this is a good sign?

argocd@argocd-repo-server-694d649776-vpg2r:~/cmp-server/config$ cd ../plugins/
argocd@argocd-repo-server-694d649776-vpg2r:~/cmp-server/plugins$ ls
avp-helm.sock

So I think overall my configuration looks correct, but the test secret deployed with argo is not updating the secret value in the k8s secret.

[werne2j]

You can’t use built in helm with a cmp.

Remove this

helm:
  valueFiles:
    - values.yaml

[razahuss02]

well, that sucks. That value consists of my overrides and its not a small file.

The deployment/configuration for AVP seems overly complex. I wanted to use this because it seemed simple enough to add an inline parameter to pull secrets from a secret store, vs an operator that will just create the secret object. Might have to look at another path.

[werne2j]

https://argocd-vault-plugin.readthedocs.io/en/stable/usage/#with-helm

You can still use a values file, just have to specify it as an env and use that in your template command.

  source:
    path: your-app
    plugin:
      name: argocd-vault-plugin-helm
      env:
        - name: HELM_ARGS
          value: -f values-dev.yaml -f values-dev-tag.yaml
          

[razahuss02]

can this also work with using multiple resources like this?

    sources:
    - repoURL: https://argoproj.github.io/argo-helm
      chart: argo-cd
      targetRevision: 5.51.6
      plugin:
        name: argocd-vault-plugin
        env:
          - name: HELM_ARGS
            value: $values/path/to/values.yaml
    - repoURL: https://gitlab.example.com/example-repo
      targetRevision: branch
      ref: values

[werne2j]

Last I checked no argoproj/argo-cd#11814 but you might want to check with Argo CD to see if that has changed