Audience claims issues when using client jwt to external vault authentication?

[Larswa]

So … trying out the Vault operator, and running through the demo in

GitHub - hashicorp-education/learn-vault-secrets-operator and the article here → The Vault Secrets Operator on Kubernetes | Vault | HashiCorp Developer

Except I am using an external Vault for these examples, and I grant the system:auth-delegator access to the service account described in the VaultAuth for my external vault Kubernetes auth. Like this

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: static-auth
  namespace: app
spec:
  method: kubernetes
  mount: kubernetes
  kubernetes:
    role: role1
    serviceAccount: default
    audiences:
    - vault

And

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-auth-delegator
subjects:
- kind: ServiceAccount
  name: default
  namespace: app
roleRef:
  kind: ClusterRole
  name: system:auth-delegator
  apiGroup: rbac.authorization.k8s.io

But this setup still gave me a 403 … untill I removed the audience from both my role definition in the Kubernetes Auth AND off course from my VaultAuth definition.

Without the audience there it just worked and whoopy… my secrets from vault were created.

Is there a known issue here with handling the audience claim when working against an a Vault external to Kubernetes?

[Larswa]

Phew ... not sure if this is indeed a bug or not. Weekend came and gone, and unable to reproduce it. Now I just get a 403 solidly all the time wit the client jwt flow. With or without audience claims, and with or without the K8s api url as a separate audience claim as reported here #379

So why it actually worked and now its not ... I spend a day trying to reproduce this with no success. Moving to New Zealand to heard sheep very very soon.

[Larswa]

Nope ... I got it working ... wrong url for kubernetes. Too long hours looking at the same problem. New Zealand is cancelled, and the issue still stands. I set it up as in the tutorial, I have to grant the default service account for app namespace the system:auth-delegator rights, and then its working. But only if there are no claims involved. The vault version running in docker is 1.15.4 and the VSO is 0.4.3

[luanaBanana]

I get the same issue. Any news here?

I'm trying with an on-prem K8s 1.28 Vanilla and with EKS 1.28.
Vault Version: 1.14.3

[Larswa]

I get the same issue. Any news here?

I'm trying with an on-prem K8s 1.28 Vanilla and with EKS 1.28. Vault Version: 1.14.3

Hi Luana,

Can you get it to work w/o using an Audience claim?

I never got around to try and intercept tokens and check it the claim was actually in there or not ... but I didnt need to use that anyway so ...

What I couldn't read from the documentation was, that I needed to add this to enable the service accounts from the applications to get a token to talk to the external vault Kubernetes Authentication method. (in this case the pods run using the default service account, so if you use a specific one, you need to set that instead)

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-auth-delegator
subjects:
  - kind: ServiceAccount
    name: default
    namespace: kube-system
  - kind: ServiceAccount
    name: default
    namespace: infrastructure
  - kind: ServiceAccount
    name: default
    namespace: default
  - kind: ServiceAccount
    name: default
    namespace: longhorn
  - kind: ServiceAccount
    name: default
    namespace: rabbitmq
  - kind: ServiceAccount
    name: default
    namespace: reflector
  - kind: ServiceAccount
    name: default
    namespace: traefik
roleRef:
  kind: ClusterRole
  name: system:auth-delegator
  apiGroup: rbac.authorization.k8s.io

[luanaBanana]

Yes, i removed the audience from the role in the auth method and from the vault CRD. This is my setup.

Vault Auth

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  annotations:
  generation: 1
  name: static-auth
  namespace: app
spec:
  kubernetes:
    role: role1
    serviceAccount: default
    tokenExpirationSeconds: 600
  method: kubernetes
  mount: demo-auth-mount
status:
  error: ""
  valid: true

Vault Static Secret

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  annotations:
  name: vault-kv-app
  namespace: app
spec:
  destination:
    create: true
    name: secretkv
    overwrite: false
  hmacSecretData: true
  mount: kvv2
  path: webapp/config
  refreshAfter: 30s
  type: kv-v2
  vaultAuthRef: static-auth

Vault Kubernetes Auth Config

vault read auth/demo-auth-mount/config

Key                               Value                                                                      
disable_iss_validation            true                                                                       
disable_local_ca_jwt              false                                                                      
issuer                                                                                                       
kubernetes_ca_cert      [REDACTED]                                           
kubernetes_host          [REDACTED]
pem_keys                          []                                                                         
token_reviewer_jwt_set            false                                                                      
use_annotations_as_alias_metadata false     

Vault Kubernetes Auth Role

Rolebinding

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
  name: vault-auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: default
  namespace: app