Allow default VaultAuth to suffix kubernetes auth role with provider/target namespace #652

mkilchhofer · 2024-03-13T13:32:00Z

Is your feature request related to a problem? Please describe.
In our current setup, our K8s platform customers use https://github.com/postfinance/vault-kubernetes to sync Secrets from Vault OSS into their desired Kubernetes namespace.

Our Vault is fully configured via Terraform. The k8s integration is done via Kubernetes Auth method on namespace basis (a Vault role per K8s namespace). In detail, we have this structure:

module "my_k8s_cluster" {
  source             = "./k8s-clusters"
  cluster_name       = "my-k8s-cluster"
  kubernetes_host    = "https://<api-server-url>"
  kubernetes_ca_cert = <<EOF
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
EOF

  disable_iss_validation = true
  reviewer_jwt           = var.my_k8s_cluster-jwt
  namespaces = {
    # Customer namespaces
    "app-a-namespace" = {
      secret_engine = "solution/my-solution-a/blabla"
      secret_path   = "general/my-app-a"
    }
    "my-app-b-namespace" = {
      secret_engine = "solution/my-solution-b/blabla"
      secret_path   = "github-actions/*"
    }
    "my-app-c-namespace" = {
      secret_engine = "solution/my-solution-c/blabla"
      secret_path   = "test/*"
    }

    # And also some platform namespaces
    "test-vault-sync" = {
      secret_engine = "realm/k8s/kv"
      secret_path   = "testing/test-vault-sync/*"
    }
    "monitoring" = {
      secret_engine = "realm/k8s/kv"
      secret_path   = "alertmanager/*"
    }
  }
}

This module then installs multiple config resources inside Vault:

########################
# Configure Vault to talk to Kubernetes
########################
resource "vault_auth_backend" "kubernetes" {
  type        = "kubernetes"
  path        = "k8s-${var.cluster_name}"
  description = "kubernetes auth for cluster ${var.cluster_name}"
}

resource "vault_kubernetes_auth_backend_config" "kubernetes" {
  backend            = vault_auth_backend.kubernetes.path
  kubernetes_host    = var.kubernetes_host
  kubernetes_ca_cert = var.kubernetes_ca_cert
  token_reviewer_jwt = var.reviewer_jwt

  disable_iss_validation = var.disable_iss_validation
}

########################
# K8s Namespace to Policy mapping
########################
resource "vault_kubernetes_auth_backend_role" "vault_sync" {
  for_each = var.namespaces

  backend                          = vault_auth_backend.kubernetes.path
  role_name                        = "vault-sync-${each.key}"                     # <-- see here
  bound_service_account_names      = ["vault-sync*"]
  bound_service_account_namespaces = [each.key]                                   # <-- here
  token_policies                   = ["k8s-${var.cluster_name}-${each.key}-read"] # <-- and here
}

########################
# Read policies per Namespace
########################
data "vault_policy_document" "k8s-policies" {
  for_each = var.namespaces

  rule {
    path         = "${each.value.secret_engine}/data/${each.value.secret_path}"
    capabilities = ["read"]
  }
  rule {
    path         = "${each.value.secret_engine}/metadata/${each.value.secret_path}"
    capabilities = ["list"]
  }
  rule {
    path         = "sys/mounts"
    capabilities = ["read"]
  }
}

resource "vault_policy" "k8s-policies" {
  for_each = var.namespaces

  name   = "k8s-${var.cluster_name}-${each.key}-read"
  policy = data.vault_policy_document.k8s-policies[each.key].hcl
}

So in practice we have:

1 vault_auth_backend per cluster
1 vault_kubernetes_auth_backend_config per cluster
N vault_policy per cluster (1 per namespace)
N vault_kubernetes_auth_backend_role per cluster (1 per namespace)

Describe the solution you'd like
It would be nice to share both

the vault default Connection
the vault default Auth (Kubernetes Auth)

with all customer namespaces, so they only need to define the VaultStaticSecret etc. :

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: default
  namespace: vault-secrets-operator                  # <-- sits in the VSO namespace
spec:
  method: kubernetes # same as tool from PF
  mount: {{ printf "k8s-%s" .Values.clusterName }}
  kubernetes:
    role: vault-sync-vso
    serviceAccount: vault-sync-vso
---
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
  name: default
  namespace: vault-secrets-operator                  # <-- sits in the VSO namespace
spec:
  address: https://vault.example.com
  caCertSecretRef: vault-ca

Describe alternatives you've considered
Only sharing the kind: VaultConnection with all namespaces and let our platform customers define the kind: VaultAuth in each namespace.

Additional context

I did a short PoC without spending too much time on the golang code which worked:

diff --git a/internal/credentials/vault/kubernetes.go b/internal/credentials/vault/kubernetes.go
index 830c1f4..0ed9af2 100644
--- a/internal/credentials/vault/kubernetes.go
+++ b/internal/credentials/vault/kubernetes.go
@@ -5,6 +5,7 @@ package vault

 import (
        "context"
+       "fmt"

        corev1 "k8s.io/api/core/v1"
        "k8s.io/apimachinery/pkg/types"
@@ -82,9 +83,11 @@ func (l *KubernetesCredentialProvider) GetCreds(ctx context.Context, client ctrl
                return nil, err
        }

+       role := fmt.Sprintf("%s-%s", l.authObj.Spec.Kubernetes.Role, l.providerNamespace)
+
        // credentials needed for Kubernetes auth
        return map[string]interface{}{
-               "role": l.authObj.Spec.Kubernetes.Role,
+               "role": role,
                "jwt":  tr.Status.Token,
        }, nil
 }

Is this a valable approach? If so, I could invest some more time to make it configurable.

Maybe somehow related:

Allow setting VaultAuth spec kubernetes mount as a default global setting #274

The text was updated successfully, but these errors were encountered:

mkilchhofer · 2024-03-18T08:06:41Z

ping @benashz

mkilchhofer added the enhancement New feature or request label Mar 13, 2024

mkilchhofer linked a pull request Apr 9, 2024 that will close this issue

feat: Add ability to add namespace as a vault role suffix #681

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow default VaultAuth to suffix kubernetes auth role with provider/target namespace #652

Allow default VaultAuth to suffix kubernetes auth role with provider/target namespace #652

mkilchhofer commented Mar 13, 2024 •

edited

mkilchhofer commented Mar 18, 2024

Allow default VaultAuth to suffix kubernetes auth role with provider/target namespace #652

Allow default VaultAuth to suffix kubernetes auth role with provider/target namespace #652

Comments

mkilchhofer commented Mar 13, 2024 • edited

mkilchhofer commented Mar 18, 2024

mkilchhofer commented Mar 13, 2024 •

edited