Vault AWS IAM Role permission denied

[andre-lx]

Describe the bug:

Trying to set one IAM Role attached to the vault service account on EKS in order to connect to S3 and KMS.

My findings:

• If we don't set any specific role with the service account, vault will use the cluster node group IAM role by default, so the cluster node group role need the S3 and KMS permissions. This works, but is not a good practice. To avoid this, we need to set a specific IAM role for the vault.

• Creating the role and attach the IAM Role ARN to the service account should be enough, but, we receive the following error:

err=
  | WebIdentityErr: failed fetching WebIdentity token:
  | caused by: WebIdentityErr: unable to read file at /var/run/secrets/eks.amazonaws.com/serviceaccount/token
  | caused by: open /var/run/secrets/eks.amazonaws.com/serviceaccount/token: permission denied

This happens because the vault command runs as the vault user (UID 100) but the container runs as root user (UID 0).

• After some searching, we found that using the security context should be enough like: Docker images with non-root account fails to read token file aws/amazon-eks-pod-identity-webhook#8

Unfortunately, setting the securityContext: runAsUser: 0 (root) the issue is the same as above, since the container already runs as root user. Setting the securityContext: runAsUser: 100 (vault), the container can't edit the vault folders permissions:

chown: /vault/config: Operation not permitted
chown: /vault/config: Operation not permitted
Could not chown /vault/config (may not have appropriate permissions)
unable to set CAP_SETFCAP effective capability: Operation not permitted

Expected behaviour:
Should be possible to use a specific IAM Role with the vault container.

Steps to reproduce the bug:
Attach some specific IAM Role to the vault service account.

Environment details:

• Kubernetes version: v1.21.5
• Cloud-provider/provisioner: EKS
• bank-vaults version: 1.14.3
• Install method: static manifests

/kind bug

[amilanoski]

I hit this exact same issue and resorted to have to deploy vault using the chart itself rather than the operator, which had some other issues. as well. Such as it does not properly support PVC #1596 and if a PVC is configured using raft HA it fails to join properly because the name IP address has changed.

[sepulworld]

Using kube2iam is not a secure path to take. I see comments to use it in code base. If running on AWS using the supported IRSA/OIDC method with service account annotation is way to go.

Use case is to use Dynamodb as a backend and appropriate IRSA/OIDC role configured to service account for Vault to use.

Getting same error noted by reporter:

2023-03-15T00:04:00.300Z [DEBUG] storage.dynamodb: adding web identity provider: roleARN=arn:aws:iam::12342342342:role/example-oidc-role
2023-03-15T00:04:00.300Z [WARN]  storage.dynamodb: error assuming role: roleARN=arn:aws:iam::12342343434:role/example-oidc-role tokenPath=/var/run/secrets/eks.amazonaws.com/serviceaccount/token sessionName=""
  err=
  | WebIdentityErr: failed fetching WebIdentity token:
  | caused by: WebIdentityErr: unable to read file at /var/run/secrets/eks.amazonaws.com/serviceaccount/token
  | caused by: open /var/run/secrets/eks.amazonaws.com/serviceaccount/token: permission denied

[github-actions]

Thank you for your contribution! This issue has been automatically marked as stale because it has no recent activity in the last 60 days. It will be closed in 20 days, if no further activity occurs. If this issue is still relevant, please leave a comment to let us know, and the stale label will be automatically removed.

[github-actions]

Thank you for your contribution! This issue has been automatically marked as stale because it has no recent activity in the last 60 days. It will be closed in 20 days, if no further activity occurs. If this issue is still relevant, please leave a comment to let us know, and the stale label will be automatically removed.