Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Apllying http-visibility breaks OAuth requests #588

Open
pandarun opened this issue Jul 14, 2021 · 2 comments
Open

Apllying http-visibility breaks OAuth requests #588

pandarun opened this issue Jul 14, 2021 · 2 comments
Labels
🐛 kind/bug This is a bug in the Hubble logic. 📊 kind/community-report This was reported by a user in the Cilium/Hubble community, eg via Slack. ⁉ needs/triage This issue requires triaging to establish the root cause.

Comments

@pandarun
Copy link

Hello, after applying http-visibility


apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: http-visibility
spec:
  endpointSelector:
    matchLabels: {}
  ingress:
    - fromEntities:
        - all
      toPorts:
        - ports:
            - port: "80"
              protocol: TCP
          rules:
            http:
              - {}
    - fromEntities:
        - all

as described in docs makes my services return 401 responses after authorization.

How can I check envoy doesn't trim Authorization headers or Cookies.

NOTE:
If I remove the policy everything works as expected.

Any help is appreciated.

@rolinh rolinh added ⁉ needs/triage This issue requires triaging to establish the root cause. 🐛 kind/bug This is a bug in the Hubble logic. 📊 kind/community-report This was reported by a user in the Cilium/Hubble community, eg via Slack. labels Jul 14, 2021
@glibsm
Copy link
Member

glibsm commented Jul 14, 2021

Is it possible your service works on more than just port 80? For example, perhaps it automatically redirects to 443 or something? It's hard to say without seeing more details of the servicec.

Using hubble observe -o json would show you the headers.

You could also print it on the server side as you process the request.

@pandarun
Copy link
Author

Is it possible your service works on more than just port 80?

You were right, one of my HTTP services was listenening on non 80 port, but it wasn't the root cause.

The root cause as I found after adding debug prints was that envoy added Content-Length: 0 header for some reason:


{
    "time": "2021-07-18T21:34:31.742188400Z",
    "verdict": "FORWARDED",
    "IP": {
        "source": "10.0.5.42",
        "destination": "10.0.3.171",
        "ipVersion": "IPv4"
    },
    "l4": {
        "TCP": {
            "source_port": 45952,
            "destination_port": 80
        }
    },
    "source": {
        "identity": 6729,
        "namespace": "default",
        "labels": [
            "k8s:app.kubernetes.io/instance=<redacted>-release",
            "k8s:app.kubernetes.io/name=<redacted>",
            "k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default",
            "k8s:io.cilium.k8s.policy.cluster=default",
            "k8s:io.cilium.k8s.policy.serviceaccount=<redacted>-release",
            "k8s:io.kubernetes.pod.namespace=default"
        ],
        "pod_name": "<redacted>-release-7699c7b884-b5r76"
    },
    "destination": {
        "ID": 1248,
        "identity": 21845,
        "namespace": "default",
        "labels": [
            "k8s:app.kubernetes.io/instance=<redacted>-release",
            "k8s:app.kubernetes.io/name=<redacted>",
            "k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=default",
            "k8s:io.cilium.k8s.policy.cluster=default",
            "k8s:io.cilium.k8s.policy.serviceaccount=<redacted>-release",
            "k8s:io.kubernetes.pod.namespace=default"
        ],
        "pod_name": "<redacted>-release-7f95449c69-qp2c4"
    },
    "Type": "L7",
    "node_name": "k3s-worker-5",
    "l7": {
        "type": "REQUEST",
        "http": {
            "method": "POST",
            "url": "http://<redacted>-release.default.svc.cluster.local/",
            "protocol": "HTTP/1.1",
            "headers": [
                {
                    "key": "Accept",
                    "value": "text/html, application/xhtml+xml, application/xml; q=0.9, image/webp, */*; q=0.8"
                },
                {
                    "key": "Accept-Encoding",
                    "value": "gzip"
                },
                {
                    "key": "Accept-Language",
                    "value": "en-US, en; q=0.5"
                },            
                **{
                    "key": "Content-Length",
                    "value": "0"
                },**
                {
                    "key": "Content-Type",
                    "value": "application/json"
                },
                {
                    "key": "Cookie",
                    "value": "<redacted>"
                },
                ...
            ]
        }
    },
    "event_type": {
        "type": 129
    },
    "traffic_direction": "INGRESS",
    "is_reply": false,
    "Summary": "HTTP/1.1 POST http://<redacted>-release.default.svc.cluster.local/"
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🐛 kind/bug This is a bug in the Hubble logic. 📊 kind/community-report This was reported by a user in the Cilium/Hubble community, eg via Slack. ⁉ needs/triage This issue requires triaging to establish the root cause.
Projects
None yet
Development

No branches or pull requests

3 participants