VSO Proxy Support

[hinrichd]

Describe the bug
When using vso on an onpremise cluster behind a proxy, vso do not use the proxy varibales from extraEnv to lookup and connect to Hashicorp Vault Secrets. Connection Test fails due to lookup on KubenDNS and not using proxy.

To Reproduce
Steps to reproduce the behavior:

1. Deploy VSO with extraEnv in helm values:
   helm install vault-secrets-operator hashicorp/vault-secrets-operator -f values.yaml

controller:
  manager:
    extraEnv:
      - name: http_proxy
        value: "http://PROXYURL:PORT"
      - name: https_proxy
        value: "http://PROXYURL:PORT"
      - name: no_proxy
        value: "127.0.0.1,localhost,.local,.svc,.svc.cluster.local"

2. Check VSO Pod Describition if proxy env is correctly applied:

Environment:
      OPERATOR_POD_NAME:          vault-secrets-operator-controller-manager-58c4dc5674-dbp75 (v1:metadata.name)
      OPERATOR_POD_UID:            (v1:metadata.uid)
      KUBERNETES_CLUSTER_DOMAIN:  cluster.local
      http_proxy:                 http://PROXYURL:PORT
      https_proxy:                http://PROXYURL:PORT
      no_proxy:                   127.0.0.1,localhost,.local,.svc,.svc.cluster.local

3. Apply Secret with SP, HCPAuth and HCPVaultSecretsApp for Test-Projekt.
4. Check VSO Pod Logs

"error": "connection check failed, err=dial tcp: lookup api.cloud.hashicorp.com on 10.200.X.X53: server misbehaving"
Lookup using internal kubedns and not using proxy, so it will fail.

2024-03-05T07:48:52Z    INFO    Starting workers        {"controller": "vaultdynamicsecret", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultDynamicSecret", "worker count": 100}
2024-03-05T07:48:52Z    ERROR   Validation failed       {"controller": "hcpauth", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "HCPAuth", "HCPAuth": {"name":"test-projekt","namespace":"vault-secrets-operator"}, "namespace": "vault-secrets-operator", "name": "test-projekt", "reconcileID": "4c8eb156-e77a-4e17-82ed-9431410eed7d", "error": "connection check failed, err=dial tcp: lookup api.cloud.hashicorp.com on 10.200.X.X:53: server misbehaving"}
github.com/hashicorp/vault-secrets-operator/controllers.(*HCPAuthReconciler).Reconcile
        /home/runner/work/vault-secrets-operator/vault-secrets-operator/controllers/hcpauth_controller.go:63
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1/pkg/internal/controller/controller.go:227
2024-03-05T07:48:52Z    ERROR   Validation failed       {"controller": "hcpauth", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "HCPAuth", "HCPAuth": {"name":"default","namespace":"vault-secrets-operator"}, "namespace": "vault-secrets-operator", "name": "default", "reconcileID": "9c399810-b5da-4109-95b5-5955acc9be92", "error": "connection check failed, err=dial tcp: lookup api.cloud.hashicorp.com on 10.200.X:X:53: server misbehaving"}
github.com/hashicorp/vault-secrets-operator/controllers.(*HCPAuthReconciler).Reconcile
        /home/runner/work/vault-secrets-operator/vault-secrets-operator/controllers/hcpauth_controller.go:63
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1/pkg/internal/controller/controller.go:227
2024-03-05T07:48:53Z    ERROR   Get App Secret  {"controller": "hcpvaultsecretsapp", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "HCPVaultSecretsApp", "HCPVaultSecretsApp": {"name":"test","namespace":"vault-secrets-operator"}, "namespace": "vault-secrets-operator", "name": "test", "reconcileID": "73167846-9b0f-4657-9116-49fdf19bf777", "appName": "test", "error": "[GET /secrets/2023-06-13/organizations/{location.organization_id}/projects/{location.project_id}/apps/{app_name}/open][403] OpenAppSecrets default  &{Code:7 Details:[] Message:}"}
github.com/hashicorp/vault-secrets-operator/controllers.(*HCPVaultSecretsAppReconciler).Reconcile
        /home/runner/work/vault-secrets-operator/vault-secrets-operator/controllers/hcpvaultsecretsapp_controller.go:121
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.1/pkg/internal/controller/controller.go:227
2024-03-05T07:48:59Z    ERROR   Get App Secret  {"controller": "hcpvaultsecretsapp", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "HCPVaultSecretsApp", "HCPVaultSecretsApp": {"name":"test","namespace":"vault-secrets-operator"}, "namespace": "vault-secrets-operator", "name": "test", "reconcileID": "dfc56edc-d055-47e4-ab26-b5069b4da7ac", "appName": "test", "error": "[GET /secrets/2023-06-13/organizations/{location.organization_id}/projects/{location.project_id}/apps/{app_name}/open][403] OpenAppSecrets default  &{Code:7 Details:[] Message:}"}

5. Deployed CURL Pod in same namespace with proxy env and and manually curled secrets from Vault Secrets to verify connection is working on cluster behind a proxy (working). Curl using lowercase http_proxy env, so also tested helm extraEnv with lower case proxy variable.

Environment

• Kubernetes version: 1.26.5
  • Distribution or cloud vendor VMware Tanzu
• vault-secrets-operator version: hashicorp/vault-secrets-operator:0.5.1

Thank You for your help.

[benashz]

Hi @hinrichd,

Thank you for the detailed report. I see two issues here.

1. The connection check on the HCPAuth resource is failing. This check is used for information purposes only, and should not affect your HCPVaultSecretsApp being synced to K8s.
2. From the HCPVaultSecretsApp logs, it seems that VSO is getting a 403 (permission denied error) when trying to fetch the Vault Secrets app. That seems like it may be related to the HCP client credentials configured on HCPAuth resource. Can you confirm that the credentials that VSO is using have the right permissions to access the HCP Vault Secrets app in question? The 403 error could also be coming from your HTTPS proxy, can you confirm that proxy URL has the correct credentials if authentication is being enforced on the proxy side?

I took a closer look at the code and did some internal testing, and it seems setting HTTPS_PROXY (lower or upper case) is being honoured. So more than likely the issue is related to the permission settings on the HCP Vault Secrets side not matching what you have in your service principal credentials. You will want to verify those credentials match the OrganizationID/ProjectID configured on your HCPAuth.

Also, just be aware that the approach you are taking by setting the proxy globally for VSO, may cause some unexpected behaviour. Ideally, VSO would support granular proxy configuration on the HCPAuth or VaultConnection custom resources. We will probably add support for that use case in a future VSO release.

Ben

[hinrichd]

Hi @benashz

Thank you for pointing out some details to my related problem. I could solve it now by changing the sp permissions.
Sounds good to have some granular proxy config on the HCP Auth in the future release.

THX
Hinrich