permission denied error on VaultStaticSecret

[ahsan-raza]

Describe the bug
I am implementing VaultStaticSecret with VSO. VaultDynamicSecret works fine with the same service account and permissions but VaultStaticSecret gives the following error:

URL: GET https://vault.vault.svc.cluster.local:8200/v1/kv-v2/data/ns/appname
Code: 403. Errors:

* 1 error occurred:
	* permission denied

	{"type": "Warning", "object": {"kind":"VaultStaticSecret","namespace":"ns","name":"vault-static-secret-appname","uid":"0d8fe0f4-5172-4897-9995-314b5e52e040","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"1758957"}, "reason": "VaultClientError"}
2024-05-07T18:33:41Z	DEBUG	events	Failed to read Vault secret: Error making API request.

To Reproduce
Steps to reproduce the behavior:

1. Deploy VSO with the following yaml file with the following VSO custom resources.

defaultVaultConnection:
  enabled: true
  address: "https://vault.vault.svc.cluster.local:8200"
  caCertSecret: "vault-ha-tls-vso"
  tlsServerName: "tls-server-vault"
  skipTLSVerify: true

• secret file:

apiVersion: v1
data:
  ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlRENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TnpFME5UYzBOelUzTUI0WERUSTBNRFV3TVRFME5EVTFOMW9YRFRNME1EUXlPVEUwTkRVMQpOMW93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRjeE5EVTNORGMxTnpCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJLdnZmekFpbVRqMEU0ZFIrbm8vMytNVURyMVZSTERRWTdiRjZyaEsKelFUMEgyRGc1Y05TckxSSS9NUU5GTFc4czF4VUZPWlVTWGZzdEhvUjB2ditQZUdqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUS9PT05JU2d1c2Y3cUZGMGJPCjhOM3dYcDRBVVRBS0JnZ3Foa2pPUFFRREFnTkhBREJFQWlBT1JRUmN2dEZEbFZYYkE2Qi9na2ZPM3BHcCtTOXkKeXVEaHovRmQ0b3Y2Z1FJZ2FVTHVqTE9wSnltOGg0ZjZWb29GQUVCeEx3TzZCWUl3UzkvWldKTTI4S2c9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
  ca.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2UUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktjd2dnU2pBZ0VBQW9JQkFRQzNqYzZ1RkxXL0Uxd2wKWDRiL2dNVWUvVVloaitmQi9wa3Jib005Z0ZIZmYwL2h5aUQrRDE0RFNOZyt4Nk9ISWQwUGV4dGJQQkl2MWZXOQpHN3ZCUjNPY1M2cFhmbmtPS2ZsWEMzNVA2V28zNWxVWG1GVElhSTA3anlwcWhlY0JtR2NadXBKdXJRSm02Wk5kCktkREV4TFRCclNYSUZlVlJpUE9QV3lnWDgzK0ovRHZQV2hjOUV6YlR1Qys5d1pYYmRFNHNWbjZGQmd0YzJzZmUKTk1Wb3NkM3pSdXFOTnJGT3BVR0FVa3FRMUxhTTNmOWt4MmNRQlhSYWlDT0NmWUdKZFEycVo5bXlka2g2bDV2TQpkSkZ3QWNBRTNiSlJlc0NjMFdoZGtFYzluNkJkRllYU1ZnelhEeWl5MDd6Yno5c2w5cnJzelRrS0tHeXFlZzQxCklXVWpWdUF0QWdNQkFBRUNnZ0VBRENydW0wRnlDSWFzNzZqM2p1cnNtNkMxc1laZVdqdGtKY244NXdxMnJWNlMKdjAzblVOTFlkdFp4Ui96SWRVTmUvekVpNVFkTys2bWtQQTNCVjhTWDRFY2FJRDdZOXF4OHpxZ2dwQVZUWGxMRgphOFpwbDdlL093WkRJbmpFdkQ2V2dSRUtOczRwdXE3bVdhdnF0QXlIeTA5TTBmTjA5ajkzbm9FYXRUY1Y5VGNzCjFIWW5aZ09RMHNYUFoxNjlnUWdCdjZLQlBnalN3NnJIRFMzRXBSUzBhNVlXNnRpclRDaWFBbjhjL2l2NzdvSmgKclVKQzlXUWVHS3hySlB2cmVGUHpuekhCbmdOdFozandnbVZnbVl6cjBqKzhrWG5CbERKZmVaMmJqT0pGZ2tXNgprbHd1U0x3OWRlOVY0N2s3emt0Snh4SUl0SnNiYlJxVERyek1Oei81R1FLQmdRRHVoR1RMdDhXL1dWVmpmTndGCkNIV3N1aGhza0NyaUluNmZ3NnpURWVMYVZZeFV4UUw0NUFucGZvc3NRcitlc2Zmb2Z6cEx0dVowWTF0Qk5sTCsKRE9qYkQzMU0zT0FDeHNlSXBMY3lJSjRIVFl0YVFEQWMvdVpMdDBmT1M3OWpjRXlhWDdIUDZXNGFXTCtSdGdaQwpNZXlxcG1vSHZIZitYYzNuMnZxbFhmaVBKUUtCZ1FERkFoRkNRbWlpNlVxN0dHa3VEc1VObGs5Y3hwU1cwUWxvCkRrMytSOUlmVnFkSUdMV3ovMDdoeTVGNS9MbnU0eE56bjUvSWtBTGpUT2c4WUh0a2ZkQnpzTHNpclhjbmZtOVMKOEJTdWVoRExuZEZqemxBSVZ0bndoUGNjTU1rRndMQnFaT2xlblVxb29BYkE1N09UdFF2cm9aVDcxRnMvQ0gxcApiUm1KbzdoaWFRS0JnQmhEbWFscndZOHN2RllKbnF4WVp2Z3lXa2U1QnRQSVpJOUdMYSt2TXgrNDhqU3hjaEwxCng5aGNDalp2ZCtUaC8vRkQrQjg1dFFvRURZVjl6RVdSOUtKTVdoZldwR2RENTRxTUR5TG5WSXd6cEVpREhTSG0KQUhHVmJKV2MrUlJabVVGZkdNeW8xNDJRbDdSd0N3VHk1VHViQUZCWEtQSFVneHppRnZ0NXFzNk5Bb0dBWmtXcQpmQUZKbkxrTGFKRkZtUGxsNXFYUFVWUnRzdFdWMG9VS2pDVHd1Z0FjRzF0b2lLYlRabmh0Ynl4NXdiLzBTeVBrCnFSUEp6QUlTMWJVb21ZU1BBR0FRNWZHelY3ZFZSM01HNllUSXowdUFkaFdXaXAyN3loYmN5YSt4eStDNk5LRnEKWFFtK3hrYzgra3ZPZDdHWEhKM25YOHhnQ2hyNE1CREpIeUQxQ1hFQ2dZRUFzQ0g0K21lNU0yNmNzSFZITk5oUQpvZ1NCR2JXOGR4MU1XT09sbTlRb3hkdUdTcHpjcGR5Tk9HZU5HNG5Id0htRE5XeW9HYkVUUEhJT3lpb2xvc0g0CkNmRTU3dldxSW9RMnlXc3piKzZqdVZ5ZlJnQjhZd1ZqZ2dpL1R3bFlJSjR0NWZWbDdDTEV6TlBucnQzSzhJUGQKelhKMnRVMWNQOER4ekFuclAyNndXTW89Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K
kind: Secret
metadata:
  creationTimestamp: null
  name: vault-ha-tls-vso
  namespace: vault

vault deployed with below values.yaml override:

global:
   enabled: true
   tlsDisable: false
injector:
   enabled: true
server:
   extraEnvironmentVars:
      VAULT_CACERT: /vault/userconfig/vault-ha-tls/vault.ca
      VAULT_TLSCERT: /vault/userconfig/vault-ha-tls/vault.crt
      VAULT_TLSKEY: /vault/userconfig/vault-ha-tls/vault.key
   volumes:
      - name: userconfig-vault-ha-tls
        secret:
         defaultMode: 420
         secretName: vault-ha-tls
   volumeMounts:
      - mountPath: /vault/userconfig/vault-ha-tls
        name: userconfig-vault-ha-tls
        readOnly: true
   standalone:
      enabled: false
   affinity: ""
   ha:
      enabled: true
      replicas: 3
      raft:
         enabled: true
         setNodeId: true
         config: |
            cluster_name = "vault-integrated-storage"
            ui = true
            listener "tcp" {
               tls_disable = 0
               address = "[::]:8200"
               cluster_address = "[::]:8201"
               tls_cert_file = "/vault/userconfig/vault-ha-tls/vault.crt"
               tls_key_file  = "/vault/userconfig/vault-ha-tls/vault.key"
               tls_client_ca_file = "/vault/userconfig/vault-ha-tls/vault.ca"
            }
            storage "raft" {
               path = "/vault/data"
            }
            disable_mlock = true
            service_registration "kubernetes" {}

• cert used for vault:

apiVersion: v1
data:
  vault.ca: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlRENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TnpFME5UYzBOelUzTUI0WERUSTBNRFV3TVRFME5EVTFOMW9YRFRNME1EUXlPVEUwTkRVMQpOMW93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRjeE5EVTNORGMxTnpCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJLdnZmekFpbVRqMEU0ZFIrbm8vMytNVURyMVZSTERRWTdiRjZyaEsKelFUMEgyRGc1Y05TckxSSS9NUU5GTFc4czF4VUZPWlVTWGZzdEhvUjB2ditQZUdqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUS9PT05JU2d1c2Y3cUZGMGJPCjhOM3dYcDRBVVRBS0JnZ3Foa2pPUFFRREFnTkhBREJFQWlBT1JRUmN2dEZEbFZYYkE2Qi9na2ZPM3BHcCtTOXkKeXVEaHovRmQ0b3Y2Z1FJZ2FVTHVqTE9wSnltOGg0ZjZWb29GQUVCeEx3TzZCWUl3UzkvWldKTTI4S2c9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
  vault.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM0akNDQW9pZ0F3SUJBZ0lRY2dKMmt6Wi96MWZuTE1ENitFSC9DREFLQmdncWhrak9QUVFEQWpBa01TSXcKSUFZRFZRUUREQmx5YTJVeUxYTmxjblpsY2kxallVQXhOekUwTlRjME56VTNNQjRYRFRJME1EVXdOakl3TXpVeApObG9YRFRJME1EZ3hOREl3TXpVeE5sb3dSekVWTUJNR0ExVUVDaE1NYzNsemRHVnRPbTV2WkdWek1TNHdMQVlEClZRUUREQ1Z6ZVhOMFpXMDZibTlrWlRvcUxuWmhkV3gwTG5OMll5NWpiSFZ6ZEdWeUxteHZZMkZzTUlJQklqQU4KQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0NBUUVBdDQzT3JoUzF2eE5jSlYrRy80REZIdjFHSVkvbgp3ZjZaSzI2RFBZQlIzMzlQNGNvZy9nOWVBMGpZUHNlamh5SGREM3NiV3p3U0w5WDF2UnU3d1Vkem5FdXFWMzU1CkRpbjVWd3QrVCtscU4rWlZGNWhVeUdpTk80OHFhb1huQVpobkdicVNicTBDWnVtVFhTblF4TVMwd2EwbHlCWGwKVVlqemoxc29GL04vaWZ3N3oxb1hQUk0yMDdndnZjR1YyM1JPTEZaK2hRWUxYTnJIM2pURmFMSGQ4MGJxalRheApUcVZCZ0ZKS2tOUzJqTjMvWk1kbkVBVjBXb2dqZ24yQmlYVU5xbWZac25aSWVwZWJ6SFNSY0FIQUJOMnlVWHJBCm5ORm9YWkJIUForZ1hSV0YwbFlNMXc4b3N0TzgyOC9iSmZhNjdNMDVDaWhzcW5vT05TRmxJMWJnTFFJREFRQUIKbzRHdE1JR3FNQTRHQTFVZER3RUIvd1FFQXdJRm9EQVRCZ05WSFNVRUREQUtCZ2dyQmdFRkJRY0RBVEFNQmdOVgpIUk1CQWY4RUFqQUFNQjhHQTFVZEl3UVlNQmFBRkQ4NDQwaEtDNngvdW9VWFJzN3czZkJlbmdCUk1GUUdBMVVkCkVRUk5NRXVDRUNvdWRtRjFiSFF0YVc1MFpYSnVZV3lDS0NvdWRtRjFiSFF0YVc1MFpYSnVZV3d1ZG1GMWJIUXUKYzNaakxtTnNkWE4wWlhJdWJHOWpZV3lDQnlvdWRtRjFiSFNIQkg4QUFBRXdDZ1lJS29aSXpqMEVBd0lEU0FBdwpSUUlnQWlqdDBSOHlSSmlDTGFIckNqWitLSUJIeEtpNVNoOGIwWmZkVzQxVVRPb0NJUUNXc2xZbjdUdllvaEh2CmJWQjAxb2hxZ1lHeTBaeXBGSUdiT3FYbXl5eW1Odz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
  vault.key: LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2UUlCQURBTkJna3Foa2lHOXcwQkFRRUZBQVNDQktjd2dnU2pBZ0VBQW9JQkFRQzNqYzZ1RkxXL0Uxd2wKWDRiL2dNVWUvVVloaitmQi9wa3Jib005Z0ZIZmYwL2h5aUQrRDE0RFNOZyt4Nk9ISWQwUGV4dGJQQkl2MWZXOQpHN3ZCUjNPY1M2cFhmbmtPS2ZsWEMzNVA2V28zNWxVWG1GVElhSTA3anlwcWhlY0JtR2NadXBKdXJRSm02Wk5kCktkREV4TFRCclNYSUZlVlJpUE9QV3lnWDgzK0ovRHZQV2hjOUV6YlR1Qys5d1pYYmRFNHNWbjZGQmd0YzJzZmUKTk1Wb3NkM3pSdXFOTnJGT3BVR0FVa3FRMUxhTTNmOWt4MmNRQlhSYWlDT0NmWUdKZFEycVo5bXlka2g2bDV2TQpkSkZ3QWNBRTNiSlJlc0NjMFdoZGtFYzluNkJkRllYU1ZnelhEeWl5MDd6Yno5c2w5cnJzelRrS0tHeXFlZzQxCklXVWpWdUF0QWdNQkFBRUNnZ0VBRENydW0wRnlDSWFzNzZqM2p1cnNtNkMxc1laZVdqdGtKY244NXdxMnJWNlMKdjAzblVOTFlkdFp4Ui96SWRVTmUvekVpNVFkTys2bWtQQTNCVjhTWDRFY2FJRDdZOXF4OHpxZ2dwQVZUWGxMRgphOFpwbDdlL093WkRJbmpFdkQ2V2dSRUtOczRwdXE3bVdhdnF0QXlIeTA5TTBmTjA5ajkzbm9FYXRUY1Y5VGNzCjFIWW5aZ09RMHNYUFoxNjlnUWdCdjZLQlBnalN3NnJIRFMzRXBSUzBhNVlXNnRpclRDaWFBbjhjL2l2NzdvSmgKclVKQzlXUWVHS3hySlB2cmVGUHpuekhCbmdOdFozandnbVZnbVl6cjBqKzhrWG5CbERKZmVaMmJqT0pGZ2tXNgprbHd1U0x3OWRlOVY0N2s3emt0Snh4SUl0SnNiYlJxVERyek1Oei81R1FLQmdRRHVoR1RMdDhXL1dWVmpmTndGCkNIV3N1aGhza0NyaUluNmZ3NnpURWVMYVZZeFV4UUw0NUFucGZvc3NRcitlc2Zmb2Z6cEx0dVowWTF0Qk5sTCsKRE9qYkQzMU0zT0FDeHNlSXBMY3lJSjRIVFl0YVFEQWMvdVpMdDBmT1M3OWpjRXlhWDdIUDZXNGFXTCtSdGdaQwpNZXlxcG1vSHZIZitYYzNuMnZxbFhmaVBKUUtCZ1FERkFoRkNRbWlpNlVxN0dHa3VEc1VObGs5Y3hwU1cwUWxvCkRrMytSOUlmVnFkSUdMV3ovMDdoeTVGNS9MbnU0eE56bjUvSWtBTGpUT2c4WUh0a2ZkQnpzTHNpclhjbmZtOVMKOEJTdWVoRExuZEZqemxBSVZ0bndoUGNjTU1rRndMQnFaT2xlblVxb29BYkE1N09UdFF2cm9aVDcxRnMvQ0gxcApiUm1KbzdoaWFRS0JnQmhEbWFscndZOHN2RllKbnF4WVp2Z3lXa2U1QnRQSVpJOUdMYSt2TXgrNDhqU3hjaEwxCng5aGNDalp2ZCtUaC8vRkQrQjg1dFFvRURZVjl6RVdSOUtKTVdoZldwR2RENTRxTUR5TG5WSXd6cEVpREhTSG0KQUhHVmJKV2MrUlJabVVGZkdNeW8xNDJRbDdSd0N3VHk1VHViQUZCWEtQSFVneHppRnZ0NXFzNk5Bb0dBWmtXcQpmQUZKbkxrTGFKRkZtUGxsNXFYUFVWUnRzdFdWMG9VS2pDVHd1Z0FjRzF0b2lLYlRabmh0Ynl4NXdiLzBTeVBrCnFSUEp6QUlTMWJVb21ZU1BBR0FRNWZHelY3ZFZSM01HNllUSXowdUFkaFdXaXAyN3loYmN5YSt4eStDNk5LRnEKWFFtK3hrYzgra3ZPZDdHWEhKM25YOHhnQ2hyNE1CREpIeUQxQ1hFQ2dZRUFzQ0g0K21lNU0yNmNzSFZITk5oUQpvZ1NCR2JXOGR4MU1XT09sbTlRb3hkdUdTcHpjcGR5Tk9HZU5HNG5Id0htRE5XeW9HYkVUUEhJT3lpb2xvc0g0CkNmRTU3dldxSW9RMnlXc3piKzZqdVZ5ZlJnQjhZd1ZqZ2dpL1R3bFlJSjR0NWZWbDdDTEV6TlBucnQzSzhJUGQKelhKMnRVMWNQOER4ekFuclAyNndXTW89Ci0tLS0tRU5EIFBSSVZBVEUgS0VZLS0tLS0K
kind: Secret
metadata:
  creationTimestamp: null
  name: vault-ha-tls
  namespace: vault

2. Any custom resources used for your secrets.
   VaultAuth.yml

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: static-auth
  namespace: ns
spec:
  method: kubernetes
  mount: kubernetes
  kubernetes:
    role: ns-role
    serviceAccount: default
    audiences:
      - vault

static-kv.yml

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  namespace: ns
  name: vault-static-secret-appname
spec:
  vaultAuthRef: static-auth
  mount: kv-v2
  type: kv-v2
  path: ns/appname

  refreshAfter: 10s
  destination:
    create: true
    name: appname
  rolloutRestartTargets:
  - kind: Deployment
    name: ef-app

4. Auth methods in Vault

vault auth enable kubernetes


vault write auth/kubernetes/config \
token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443 \
kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
issuer="https://kubernetes.default.svc.cluster.local"

5. Roles, policies, in vault

vault secrets enable -version=2 kv-v2

vault policy write static-secret - << EOF
    path "kv-v2/data/ns/appname " {
    capabilities = ["read"]
    }
EOF

vault write auth/kubernetes/role/ns-role \
bound_service_account_names=default \
bound_service_account_namespaces=ns \
policies=default,static-secret \
audience=vault \
ttl=24h

6. Error: (Retrieved from logs of vso pod)

URL: GET https://vault.vault.svc.cluster.local:8200/v1/kv-v2/data/ns/appname
Code: 403. Errors:

* 1 error occurred:
	* permission denied

	{"type": "Warning", "object": {"kind":"VaultStaticSecret","namespace":"ns","name":"vault-static-secret-appname","uid":"0d8fe0f4-5172-4897-9995-314b5e52e040","apiVersion":"secrets.hashicorp.com/v1beta1","resourceVersion":"1758957"}, "reason": "VaultClientError"}
2024-05-07T18:33:41Z	DEBUG	events	Failed to read Vault secret: Error making API request.

Application deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: ef-app
  name: ef-app
  namespace: ns
spec:
  selector:
    matchLabels:
      app: ef-app
  template:
    metadata:
      labels:
        app: ef-app
    spec:
      containers:
        - image: nginx
          name: nginx
          env:
          - name: SECRET_USERNAME
            valueFrom:
              secretKeyRef:
                name: appname
                key: username
          - name: SECRET_PASSWORD
            valueFrom:
              secretKeyRef:
                name: appname
                key: password
          - name: URI_TOKEN
            valueFrom:
              secretKeyRef:
                name: appname
                key: token
          resources: {}

kubectl describe VaultStaticSecret output.

Events:
  Type     Reason            Age                   From               Message
  ----     ------            ----                  ----               -------
  Warning  VaultClientError  74s (x25 over 2m52s)  VaultStaticSecret  Failed to read Vault secret: Error making API request.

URL: GET https://vault.vault.svc.cluster.local:8200/v1/kv-v2/data/ns/appname
Code: 403. Errors:

* 1 error occurred:
  * permission denied

Expected behavior
VaultStaticSecret should create a kv secret in the namespace and inject in the pod.

Environment

• Ubuntu server 22-04
• Kubernetes version:
  • v1.28.9+rke2r1
• vault-secrets-operator version: 0.6.0
• vault version: 1.16.1

Additional context

I followed this guide to deploy the vault
https://developer.hashicorp.com/vault/tutorials/kubernetes/kubernetes-minikube-tls

Firstly, I assumed its the problem with tls so I deployed vault without tls and still the VSO was not able to create Static Secrets but was able to create dynamic secrets.

[asttle]

I have faced similar issue Reconciler error. Try to disable and re-enable kubernetes auth in vault server and configure and add the role and try again.

It worked for me.

[ahsan-raza]

@asttle Thanks for your comment and suggestion, I followed your steps, unfortunately, it did not work for me. I have deployed vault and 2 different server and also on minikube on my mac, I am facing the same issue