kubeseal is ignoring NO_PROXY variables upon request for cert.pem (403)

[kriipke]

Which component:
version 0.25 of sealed secrets, tried with both 0.25 and 0.26 of the kubeseal client

Describe the bug

kubeseal is ignoring NO_PROXY variables upon request to:
https://10.20.157.251:6443/api/v1/namespaces/sealed-secrets/services/http:eksa-dev-sealed-secrets:http/proxy/v1/cert.pem

Other workloads on the cluster are using the no_proxy vars just fine so we know they're set correctly on the nodes. Any thoughts?

To Reproduce
Steps to reproduce the behavior:

cat /tmp/flame.yaml  | kubeseal --controller-name eksa-dev-sealed-secrets --controller-namespace sealed-secrets -v9
I0307 16:48:30.122752    9784 loader.go:395] Config loaded from file:  /home/spencer/.kube/config.new
I0307 16:48:30.124765    9784 round_trippers.go:466] curl -v -XGET  -H "Accept: application/x-pem-file, */*" -H "User-Agent: kubeseal/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://10.20.157.251:6443/api/v1/namespaces/sealed-secrets/services/eksa-dev-sealed-secrets'
I0307 16:48:30.138387    9784 round_trippers.go:510] HTTP Trace: Dial to tcp:10.20.157.251:6443 succeed
I0307 16:48:30.181576    9784 round_trippers.go:553] GET https://10.20.157.251:6443/api/v1/namespaces/sealed-secrets/services/eksa-dev-sealed-secrets 200 OK in 56 milliseconds
I0307 16:48:30.181642    9784 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 13 ms TLSHandshake 15 ms ServerProcessing 27 ms Duration 56 ms
I0307 16:48:30.181648    9784 round_trippers.go:577] Response Headers:
I0307 16:48:30.181679    9784 round_trippers.go:580]     Cache-Control: no-cache, private
I0307 16:48:30.181713    9784 round_trippers.go:580]     Content-Type: application/json
I0307 16:48:30.181717    9784 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: e1f30f59-a886-4c1e-87d7-0fe2b1babbe2
I0307 16:48:30.181732    9784 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: c4fea2d7-140b-4b07-819f-f43ed6866ef4
I0307 16:48:30.181747    9784 round_trippers.go:580]     Content-Length: 2445
I0307 16:48:30.181750    9784 round_trippers.go:580]     Date: Thu, 07 Mar 2024 21:48:29 GMT
I0307 16:48:30.181752    9784 round_trippers.go:580]     Audit-Id: 5a2cebf6-d50f-4e48-b356-86189188b0bf
I0307 16:48:30.181834    9784 request.go:1212] Response Body: {"kind":"Service","apiVersion":"v1","metadata":{"name":"eksa-dev-sealed-secrets","namespace":"sealed-secrets","uid":"17616c06-db33-4efc-9b9d-827465e8c4ee","resourceVersion":"2770070","creationTimestamp":"2024-03-07T20:14:19Z","labels":{"app.kubernetes.io/instance":"release-name","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"sealed-secrets","app.kubernetes.io/part-of":"sealed-secrets","app.kubernetes.io/version":"0.25.0","argocd.argoproj.io/instance":"sealedsecrets-devdw01","helm.sh/chart":"sealed-secrets-2.14.2"},"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"labels\":{\"app.kubernetes.io/instance\":\"release-name\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"sealed-secrets\",\"app.kubernetes.io/part-of\":\"sealed-secrets\",\"app.kubernetes.io/version\":\"0.25.0\",\"argocd.argoproj.io/instance\":\"sealedsecrets-devdw01\",\"helm.sh/chart\":\"sealed-secrets-2.14.2\"},\"name\":\"eksa-dev-sealed-secrets\",\"namespace\":\"sealed-secrets\"},\"spec\":{\"ports\":[{\"name\":\"http\",\"nodePort\":null,\"port\":8080,\"targetPort\":\"http\"}],\"selector\":{\"app.kubernetes.io/instance\":\"release-name\",\"app.kubernetes.io/name\":\"sealed-secrets\"},\"type\":\"ClusterIP\"}}\n"},"managedFields":[{"manager":"argocd-controller","operation":"Update","apiVersion":"v1","time":"2024-03-07T20:14:19Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}},"f:labels":{".":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/name":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:argocd.argoproj.io/instance":{},"f:helm.sh/chart":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":8080,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}}}]},"spec":{"ports":[{"name":"http","protocol":"TCP","port":8080,"targetPort":"http"}],"selector":{"app.kubernetes.io/instance":"release-name","app.kubernetes.io/name":"sealed-secrets"},"clusterIP":"10.180.172.15","clusterIPs":["10.180.172.15"],"type":"ClusterIP","sessionAffinity":"None","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","internalTrafficPolicy":"Cluster"},"status":{"loadBalancer":{}}}
I0307 16:48:30.182524    9784 round_trippers.go:466] curl -v -XGET  -H "Accept: application/x-pem-file, */*" -H "User-Agent: kubeseal/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://10.20.157.251:6443/api/v1/namespaces/sealed-secrets/services/http:eksa-dev-sealed-secrets:http/proxy/v1/cert.pem'
I0307 16:48:30.198596    9784 round_trippers.go:553] GET https://10.20.157.251:6443/api/v1/namespaces/sealed-secrets/services/http:eksa-dev-sealed-secrets:http/proxy/v1/cert.pem 403 Forbidden in 15 milliseconds     
I0307 16:48:30.198667    9784 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 15 ms Duration 15 ms
I0307 16:48:30.198672    9784 round_trippers.go:577] Response Headers:
I0307 16:48:30.198696    9784 round_trippers.go:580]     Vary: Accept-Language
I0307 16:48:30.198700    9784 round_trippers.go:580]     Via: 1.1 devqa-proxy01 (squid/4.10)
I0307 16:48:30.198702    9784 round_trippers.go:580]     X-Cache: MISS from devqa-proxy01
I0307 16:48:30.198704    9784 round_trippers.go:580]     Audit-Id: db7f2df2-55b8-4c59-b771-974118a5a706
I0307 16:48:30.198706    9784 round_trippers.go:580]     Content-Language: en
I0307 16:48:30.198708    9784 round_trippers.go:580]     Content-Type: text/html;charset=utf-8
I0307 16:48:30.198711    9784 round_trippers.go:580]     Mime-Version: 1.0
I0307 16:48:30.198726    9784 round_trippers.go:580]     Server: squid/4.10
I0307 16:48:30.198728    9784 round_trippers.go:580]     X-Squid-Error: ERR_ACCESS_DENIED 0
I0307 16:48:30.198732    9784 round_trippers.go:580]     Cache-Control: no-cache, private
I0307 16:48:30.198734    9784 round_trippers.go:580]     Date: Thu, 07 Mar 2024 21:48:29 GMT
I0307 16:48:30.198735    9784 round_trippers.go:580]     X-Cache-Lookup: NONE from devqa-proxy01:3128
I0307 16:48:30.198737    9784 round_trippers.go:580]     Content-Length: 3902

You'll notice the X-Squid-Error: ERR_ACCESS_DENIED 0 in the response headers on the request to https://10.20.157.251:6443/api/v1/namespaces/sealed-secrets/services/http:eksa-dev-sealed-secrets:http/proxy/v1/cert.pem.

Below are the settings we have set for no_proxy and NO_PROXY we have set on every Kubernetes node:

Expected behavior
A clear and concise description of what you expected to happen.

cat /tmp/secret.yaml  | kubeseal --controller-name eksa-dev-sealed-secrets --controller-namespace sealed-secrets -v9
I0307 16:39:45.619156    6205 loader.go:395] Config loaded from file:  /home/spencer/.kube/config.new
I0307 16:39:45.621467    6205 round_trippers.go:466] curl -v -XGET  -H "Accept: application/x-pem-file, */*" -H "User-Agent: kubeseal/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://10.20.156.251:6443/api/v1/namespaces/sealed-secrets/services/eksa-dev-sealed-secrets'
I0307 16:39:45.636424    6205 round_trippers.go:510] HTTP Trace: Dial to tcp:10.20.156.251:6443 succeed
I0307 16:39:45.702002    6205 round_trippers.go:553] GET https://10.20.156.251:6443/api/v1/namespaces/sealed-secrets/services/eksa-dev-sealed-secrets 200 OK in 80 milliseconds
I0307 16:39:45.702038    6205 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 14 ms TLSHandshake 51 ms ServerProcessing 13 ms Duration 80 ms
I0307 16:39:45.702042    6205 round_trippers.go:577] Response Headers:
I0307 16:39:45.702047    6205 round_trippers.go:580]     Cache-Control: no-cache, private
I0307 16:39:45.702049    6205 round_trippers.go:580]     Content-Type: application/json
I0307 16:39:45.702051    6205 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: 026c84a0-5672-439c-bd1b-d124a0c84a2b
I0307 16:39:45.702053    6205 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 946d256b-75a1-4a03-ade4-d33c50fdb1ff
I0307 16:39:45.702055    6205 round_trippers.go:580]     Content-Length: 2443
I0307 16:39:45.702056    6205 round_trippers.go:580]     Date: Thu, 07 Mar 2024 21:39:41 GMT
I0307 16:39:45.702058    6205 round_trippers.go:580]     Audit-Id: 3c66040d-9424-4fa5-a34a-0e186a82e9f8
I0307 16:39:45.702091    6205 request.go:1212] Response Body: {"kind":"Service","apiVersion":"v1","metadata":{"name":"eksa-dev-sealed-secrets","namespace":"sealed-secrets","uid":"84672da6-aff6-4912-a117-2712d93c37cd","resourceVersion":"2028736","creationTimestamp":"2024-02-28T19:40:10Z","labels":{"app.kubernetes.io/instance":"release-name","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"sealed-secrets","app.kubernetes.io/part-of":"sealed-secrets","app.kubernetes.io/version":"0.25.0","argocd.argoproj.io/instance":"sealedsecrets-devpw01","helm.sh/chart":"sealed-secrets-2.14.2"},"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"labels\":{\"app.kubernetes.io/instance\":\"release-name\",\"app.kubernetes.io/managed-by\":\"Helm\",\"app.kubernetes.io/name\":\"sealed-secrets\",\"app.kubernetes.io/part-of\":\"sealed-secrets\",\"app.kubernetes.io/version\":\"0.25.0\",\"argocd.argoproj.io/instance\":\"sealedsecrets-devpw01\",\"helm.sh/chart\":\"sealed-secrets-2.14.2\"},\"name\":\"eksa-dev-sealed-secrets\",\"namespace\":\"sealed-secrets\"},\"spec\":{\"ports\":[{\"name\":\"http\",\"nodePort\":null,\"port\":8080,\"targetPort\":\"http\"}],\"selector\":{\"app.kubernetes.io/instance\":\"release-name\",\"app.kubernetes.io/name\":\"sealed-secrets\"},\"type\":\"ClusterIP\"}}\n"},"managedFields":[{"manager":"argocd-controller","operation":"Update","apiVersion":"v1","time":"2024-02-28T19:40:10Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}},"f:labels":{".":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:app.kubernetes.io/name":{},"f:app.kubernetes.io/part-of":{},"f:app.kubernetes.io/version":{},"f:argocd.argoproj.io/instance":{},"f:helm.sh/chart":{}}},"f:spec":{"f:internalTrafficPolicy":{},"f:ports":{".":{},"k:{\"port\":8080,\"protocol\":\"TCP\"}":{".":{},"f:name":{},"f:port":{},"f:protocol":{},"f:targetPort":{}}},"f:selector":{},"f:sessionAffinity":{},"f:type":{}}}}]},"spec":{"ports":[{"name":"http","protocol":"TCP","port":8080,"targetPort":"http"}],"selector":{"app.kubernetes.io/instance":"release-name","app.kubernetes.io/name":"sealed-secrets"},"clusterIP":"10.180.49.50","clusterIPs":["10.180.49.50"],"type":"ClusterIP","sessionAffinity":"None","ipFamilies":["IPv4"],"ipFamilyPolicy":"SingleStack","internalTrafficPolicy":"Cluster"},"status":{"loadBalancer":{}}}
I0307 16:39:45.702484    6205 round_trippers.go:466] curl -v -XGET  -H "Accept: application/x-pem-file, */*" -H "User-Agent: kubeseal/v0.0.0 (linux/amd64) kubernetes/$Format" 'https://10.20.156.251:6443/api/v1/namespaces/sealed-secrets/services/http:eksa-dev-sealed-secrets:http/proxy/v1/cert.pem'
I0307 16:39:45.720084    6205 round_trippers.go:553] GET https://10.20.156.251:6443/api/v1/namespaces/sealed-secrets/services/http:eksa-dev-sealed-secrets:http/proxy/v1/cert.pem 200 OK in 17 milliseconds
I0307 16:39:45.720125    6205 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 17 ms Duration 17 ms
I0307 16:39:45.720129    6205 round_trippers.go:577] Response Headers:
I0307 16:39:45.720144    6205 round_trippers.go:580]     Date: Thu, 07 Mar 2024 21:39:35 GMT
I0307 16:39:45.720157    6205 round_trippers.go:580]     Content-Length: 1724
I0307 16:39:45.720159    6205 round_trippers.go:580]     Audit-Id: 4c45100f-aaf0-4b4c-9123-6f58342eacdf
I0307 16:39:45.720172    6205 round_trippers.go:580]     Cache-Control: no-cache, private
I0307 16:39:45.720184    6205 round_trippers.go:580]     Content-Type: application/x-pem-file
{
  "kind": "SealedSecret",
  "apiVersion": "bitnami.com/v1alpha1",
  "metadata": {
    "name": "flame-vars",
    "namespace": "default",
    "creationTimestamp": null
  },
  "spec": {
    "template": {
      "metadata": {
        "name": "flame-vars",
        "namespace": "default",
        "creationTimestamp": null
      }
    },
    "encryptedData": {
      "PASSWORD": "AgAX0yRADabrY1lb51sWFXgVpeMeSrvsmY/BukWOeHdyNZ+f+FB6goUNDO59WariGxLkWr7s7tnrInjsnvQ2/cpVkHGdgcVNgwZSiuz+dyOm2cK6RTeF5/2T/O+2XMKQMIx5JzFk9dnsBGuOrLAlWwJ6F0qBkgCkLH/iGdXbUYlpUst7UjUi2PXlI4hOtQVpipOE3/Dr+WFES8/b0Pqu9lScniBOajaebWQRdw99wACK+CPALPtCNP7XF6ffXU4/uCilOu732GeZERjHB8JFQXcoFt6zJhz+9LKmt9qfXZCLi2lQLVAnZ/Qr+rYjDTaly9uLWMqXrXODoxherWWGZWDxhYEQ2ayfblbe+0XK9bKa+ei57jUkNRlYUqpQs2lEuWHeYx+HlVPRpijOthSQF8ZcYwrCGz7n3Vf+PEdIbFJdD06sK8nYa/zKZd4nUbVKzehHMcdgsTyR5HbYON9JjiRA63t6lea5ploDKswo9RxQpFX3/chyfggJp0guAbpUuT9XHinnK8kdpbFANmn5UThiDuyuM3TuQDd7GqBlHuzUVGsrVCsGy5axwIkdn3mlCOVNcvi0kOMgXX1E6JZx3azUgA7fawihKAWPXuhAPIGHJduKdaK2OQ0CkJlKEGZ+Eg4KuRZ1VNI3KPC3+CfGaXT97KyDzozBNIq82EEL3KdgWBgoTqeRpMJ9i3dNg8aesRXMnL4tWxOMOA=="
    }
  }
}

Version of Kubernetes:

• Output of kubectl version:

Client Version: v1.27.6-eks-aab2b08
Kustomize Version: v5.0.1
Server Version: v1.27.4-eks-cedffd4

Additional context

[alemorcuq]

Do you have the NO_PROXY variable set on your local environment? I assume you are using kubeseal from your local environment and not from inside the cluster, so kubeseal will get the proxy settings from your local environment.

[kriipke]

It is set, however, the machine I'm on shouldn't be using the proxy at all..

The reason we shared the cluster info is that the same kubeseal client will work on one of our clusters but not on the one I'm posting about, so we figured it must be a difference in how the clusters are configured. However, when we look at the sealed-secrets-controller logs it looks like the kubeseal requests aren't even making it to the cluster because they're going through the proxy.

Super weird?

[alemorcuq]

It is indeed weird, because I see two requests GET requests in your logs but the first one is succeeding, so it seems the NO_PROXY variable is functioning correctly there.

Can you try with an older version so we can see if there's something different? For example, 0.24.5.

[kriipke]

yeah! it is weird, we saw the exact same thing, we're not sure why the second request is all of a sudden ignoring the NO_PROXY var.

We will give 0.24.5 a try, thanks!