Can't create mongodb secret-engine

[BryanDollery]

Describe the bug:
I want to use the configurer to create a db secrets engine (MongoDB) for dynamic creds.

Expected behaviour:
I expect a db engine called "mongo" to be created with a set of roles that can be used to dynamically create creds.

Steps to reproduce the bug:
I have installed vault using the bank-vaults operator in my security namespace. Here's my config:

apiVersion: "vault.banzaicloud.com/v1alpha1"
kind: "Vault"
metadata:
  name: "vault"
  namespace: security
spec:
  size: 1
  image: vault:latest
  annotations:
    common/annotation: "true"
  vaultAnnotations:
    type/instance: "vault"
  vaultConfigurerAnnotations:
    type/instance: "vaultconfigurer"
  vaultLabels:
    example.com/log-format: "json"
  vaultConfigurerLabels:
    example.com/log-format: "json"
  serviceAccount: vault
  serviceType: ClusterIP
  volumes:
    - name: vault-file
      persistentVolumeClaim:
        claimName: vault-file
  volumeMounts:
    - name: vault-file
      mountPath: /vault/file
  caNamespaces:
    - "security"
  unsealConfig:
    options:
      preFlightChecks: true
      storeRootToken: true
    kubernetes:
      secretNamespace: security
  config:
    storage:
      file:
        path: "${ .Env.VAULT_STORAGE_FILE }"
    listener:
      tcp:
        address: "0.0.0.0:8200"
        tls_disable: false
        tls_cert_file: /vault/tls/server.crt
        tls_key_file: /vault/tls/server.key
    telemetry:
      statsd_address: localhost:9125
    ui: true
  externalConfig:
    policies:
      - name: allow_mongo
        rules: path "mongo/*" { capabilities = ["read", "update", "list"] }
      - name: allow_secrets
        rules: path "secret/*" { capabilities = ["create", "read", "update", "delete", "list"] }
    groups:
      - name: admin1
        policies:
          - allow_secrets
        metadata:
          privileged: true
        type: external
      - name: admin2
        policies:
          - allow_secrets
        metadata:
          privileged: true
        type: external
    group-aliases:
      - name: admin1
        mountpath: token
        group: admin1
    auth:
      - type: kubernetes
        roles:
          - name: default
            bound_service_account_names: ["default", "vault-secrets-webhook", "vault"]
            bound_service_account_namespaces: ["default", "platform", "ctx-service-invoice", "automation", "security", "test"]
            policies: ["allow_secrets", "allow_pki", "allow_mongo"]
            ttl: 1h
    secrets:
      - path: secret
        type: kv
        description: General secrets.
        options:
          version: 2
      - type: database
        description: MongoDB for nga with mongo access
        configuration:
          config:
            - name: mongo
              plugin_name: "mongodb-database-plugin"
              connection_url: "mongodb+srv://{{username}}:{{password}}@nga-mongodb-svc.db/admin?ssl=false"
              allowed_roles: ["api-r", "api-w", "api-rw", "entity-rw", "webhooks-r", "webhooks-w", "godzilla"]
              username: "vault"
              password: "This.can.be.literally.anything.really."
          roles:
            - name: "api-r"
              db_name: "mongo"
              creation_statements: "{ 'db': 'admin', 'roles': [ {'role': 'read', 'db': 'service-invoice-api'}] }"
              default_ttl: "10m"
              max_ttl: "24h"
  vaultEnvsConfig:
    - name: VAULT_LOG_LEVEL
      value: debug
    - name: VAULT_STORAGE_FILE
      value: "/vault/file"
  istioEnabled: false
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: vault-file
  namespace: security
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  hostPath:
    path: /vault/file
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vault-file
  namespace: security
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi

I think that's a pretty normal config. When I apply it I see vault, the UI works fine and I can log in and look around, and in the UI I can see the policies and the /secrets kv engine have all been created properly as per the config. I see no problems in connectivity.

What I do see is an error in the config pod:

{"level":"info","msg":"adding secret engine  (database)","time":"2022-05-08T08:31:17Z"}
{"level":"error","msg":"error configuring vault: error configuring secret engines for vault: error adding secrets engines: error mounting  into vault: Error making API request.\n\nURL: POST https://vault.security:8200/v1/sys/mounts\nCode: 405. Errors:\n\n* 1 error occurred:\n\t* unsupported operation\n\n","time":"2022-05-08T08:31:17Z"}
{"level":"info","msg":"Failed applying configuration file: /config/vault-configurer/vault-config.yml , sleeping for 1m0s before trying again","time":"2022-05-08T08:31:17Z"}

At the core of which is Unsupported Operation. I see no other errors in any other container in any other pod. I can see that the error is coming from vault itself. I have tried the same thing without the roles to narrow down the error, but I get the same result.

It might be worth noting that I can create the db with the vault cli like this:

vault secrets enable -path mongo database

vault write mongo/config/nga \
  plugin_name=mongodb-database-plugin \
  allowed_roles="vault,api-r,api-w,api-rw,entity-rw,webhooks-r,webhooks-w" \
  connection_url="$$(kubectl get secret mongodb -o json | jq '.data."connectionString.standardSrv"' | xargs | base64 -d)" \
  username="$$(kubectl get secret mongodb -o jsonpath='{.data.username}' | base64 -d)" \
  password="$$(kubectl get secret mongodb -o jsonpath='{.data.password}'  | base64 -d)"

vault write mongo/roles/api-r \
  db_name=nga \
  creation_statements='{ "db": "admin", "roles": [ {"role": "read", "db": "service-invoice-api"}] }' \
  default_ttl="5m" \ 
  max_ttl="10m"

Environment details:

• Kubernetes version: v1.22.4
• Cloud-provider/provisioner: AKS
• bank-vaults version: latest at time of writing
• Install method: helm for the operator

/kind bug

[BryanDollery]

Actually, it gets stranger. I'm having a similar problem configuring PKI now:

    secrets:
      - type: pki
        description: Vault PKI Backend
        config:
          default_lease_ttl: 168h
          max_lease_ttl: 720h
        configuration:
          config:
            - name: urls
              issuing_certificates: https://vault-0.secuity:8200/v1/pki/ca
              crl_distribution_points: https://vault-0.security:8200/v1/pki/crl
          root/generate:
            - name: exported
              common_name: vault.vault
              create_only: true
              save_to: "secret/data/pki/ca"
          roles:
            - name: default
              allowed_domains: localhost,pod,svc,default
              allow_subdomains: true
              generate_lease: true
              ttl: 30m

And this results in:

{"level":"info","msg":"adding secret engine  (pki)","time":"2022-05-09T08:15:08Z"}
{"level":"error","msg":"error configuring vault: error configuring secret engines for vault: error adding secrets engines: error mounting  into vault: Error making API request.\n\nURL: POST https://vault.security:8200/v1/sys/mounts\nCode: 405. Errors:\n\n* 1 error occurred:\n\t* unsupported operation\n\n","time":"2022-05-09T08:15:08Z"}

But,

    secrets:
    - path: secret
        type: kv
        description: General secrets.
        options:
          version: 2

still works fine and I can create the engine through the UI without issue

[dmolik]

I'm having the same issue with pki, I'd be happy if I was getting better logging

[dmolik]

Okay turns out I needed to add path key to my pki block there are a few other open related tickets

[BryanDollery]

@dmolik Can you post an example, pls?

[notjames]

one thing to note here (bumped into this as I was attempting to find out how to enable the use of VSW with the mongodb plugin) is that the /secrets API endpoint is not where the database for mongodb is enabled. The mongodb endpoint will be where the -path argument specifies it to be and if not specified then the endpoint is /database/mongodb. It appears that the OP was at least attempting to test this using the /secrets endpoint. Now I'm not sure this isn't still a bug as I'm not able to get the operator to create the endpoint (yet) as far as I can tell.