Unable to inject secrets in to my pod

[tamipangadil]

Hi, I'm trying to inject secrets from Vault server to my pod and it always say:

[WARN] (view) vault.read(secret/data/dev/my-secret): no secret exists at secret/data/dev/my-secret (retry attempt 9 after "1m0s")

I've already added the Vault policy to secret

path "secret/*" {
  capabilities = ["read"]
}

The annoying part is that it works when I put it outside /data/.

I can read /secret/my-secret and able to inject it smoothly.

Is this a bug or I'm just doing something not correct?

[carlosjgp]

KV2 backend required the data bit on the path, it's on the documentation.

You are also referring to two different paths... One with dev and the other without it.

Adding a first with the commands and files to reproduce your problem could help other people to help you

[tamipangadil]

Thanks @carlosjgp. I understand the role of data path on KV2.

Here's the Kubernetes annotations that I'm trying to use to pull from secret/data

        ...
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-secret-my-secret: "secret/data/dev/my-secret"
        vault.hashicorp.com/agent-inject-template-my-secret: |
          {{- with secret "secret/data/dev/my-secret" -}}
            {{- range $k, $v := .Data.data }}
              {{ $k }}="{{ $v }}"
            {{- end }}
          {{- end }}
        vault.hashicorp.com/role: "hello-world"
        ...

The annotations above will not pull the secrets from secret/data/ and will always throw error:

[WARN] (view) vault.read(secret/data/dev/my-secret): no secret exists at secret/data/dev/my-secret (retry attempt 9 after "1m0s")

However, it will work if I take it out from secret/data folder.

[carlosjgp]

Thanks @carlosjgp. I understand the role of data path on KV2.

Here's the Kubernetes annotations that I'm trying to use to pull from secret/data

        ...
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-secret-my-secret: "secret/data/dev/my-secret"
        vault.hashicorp.com/agent-inject-template-my-secret: |
          {{- with secret "secret/data/dev/my-secret" -}}
            {{- range $k, $v := .Data.data }}
              {{ $k }}="{{ $v }}"
            {{- end }}
          {{- end }}
        vault.hashicorp.com/role: "hello-world"
        ...

The annotations above will not pull the secrets from secret/data/ and will always throw error:

[WARN] (view) vault.read(secret/data/dev/my-secret): no secret exists at secret/data/dev/my-secret (retry attempt 9 after "1m0s")

However, it will work if I take it out from secret/data folder.

@tamipangadil

yeah KV2 backend is a little bit confusing to use but you don't need data on your path for the consul template (vault.hashicorp.com/agent-inject-template-my-secret)
but you need data on your path when you are creating a policy

the secret path should be the same as with the CLI

$ vault login ...

$ vault kv get secret/dev/my-secret

If the commands on the CLI work it should work with the sidecar as far as the logins use the same policies/ACLs

[darkedges]

Okay still confused. Using Vault 1.15.2

Here is my template

{{ with secret "localdev/kv/consentui/nginx/certificate" }}{{ .[Data.data.Ca](http://data.data.ca/) | base64Decode }}{{ end }}

Here is the log in Vault Agent

2024-02-13T23:12:48.050Z [TRACE] (view) vault.read(localdev/kv/consentui/nginx/certificate) starting fetch
2024-02-13T23:12:48.050Z [TRACE] vault.read(localdev/kv/consentui/nginx/certificate): GET /v1/localdev/kv/consentui/data/nginx/certificate
2024-02-13T23:12:48.056Z [WARN] (view) vault.read(localdev/kv/consentui/nginx/certificate): no secret exists at localdev/kv/consentui/data/nginx/certificate (retry attempt 10 after "1m0s")

Here is the log in Vault

2024-02-13T23:12:48.056Z [TRACE] core: completed_request: start_time=2024-02-13T23:12:48Z duration=1ms client_id=e830b4f1-2403-a044-6015-3b8e2ee55ba3 client_address=10.1.2.77:53136 status_code=200 request_path=/v1/localdev/kv/consentui/data/nginx/certificate request_method=GET

Here is my policy

path "localdev/kv/consentui/*" {
  capabilities = ["read","list"]
}

Here is a request using a token from that policy

curl  --header "X-Vault-Token: hvs.CAESIMQ5Qo9nHG2ZOe0cYQ5UhYl02iQ9bqksNVfY5fSUHJZsGh4KHGh2cy5MajZkaGNDSDRhblRpallDWmFXTHIyeE4"  --request GET https://vault.darkedges.localhost/v1/localdev/kv/consentui/data/nginx/certificate

and the response

{
  "request_id": "327fabef-51e4-2e84-19ef-73e95c5b8bd0",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": {
    "data": {
      "Ca": "xxx",
      "Cert": "xxx",
      "Key": "xxx"
    },
    "metadata": {
      "created_time": "2024-02-13T23:03:24.381226889Z",
      "custom_metadata": null,
      "deletion_time": "2024-02-14T02:33:24.381226889Z",
      "destroyed": false,
      "version": 1
    }
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}

[darkedges]

What I have found via debugging vault is that if I create a secret with an expiry time the code check does not validate that it has expired and sets the secret return as null. Once I removed that deletion time it started to work. Will raise an issue with vault directly.

hashicorp/consul-template#1881