Cilium drops traffic without Hubble showing it

[Pionerd]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

We applied CiliumNetworkPolicies to scope down the traffic for the External Secrets Operator (ESO), as described here: https://external-secrets.io/v0.8.5/guides/security-best-practices/#network-policy. These CNPs (please see the applicable ones below) function perfectly on EKS, AKS and GKE. However, on RKE2 clusters they result in a broken External Secrets Operator.

The ESO is not logging anything useful, but neither is Hubble. hubble observe -f --verdict DROPPED gives me literally nothing. Removing the CNPs resolves the issue.

(Little bit extra background ESO specific: what is extra weird is that existing secrets stay in SecretSynced status, as if everything is OK. Removing a secret and adding it again brings the issue to light, but without ESO throwing any meaningful error. Only when the Azure Workload Identity expires we see DNS lookups to login.microsoftonline.com failing, even though they are specifically allowed (see below)).

Cilium Version

Client: 1.15.1 a368c8f 2024-02-14T22:16:57+00:00 go version go1.21.6 linux/amd64
Daemon: 1.15.1 a368c8f 2024-02-14T22:16:57+00:00 go version go1.21.6 linux/amd64

Kernel Version

Linux 5.15.0-102-generic #112-Ubuntu SMP Tue Mar 5 16:50:32 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Kubernetes Version

v1.28.8+rke2r1

Regression

No response

Sysdump

cilium-sysdump-20240422-141909.zip

Relevant log output

No response

Anything else?

---
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"cilium.io/v2","kind":"CiliumNetworkPolicy","metadata":{"annotations":{},"labels":{"argocd.argoproj.io/instance":"external-secrets"},"name":"default","namespace":"external-secrets"},"spec":{"egress":[{"toPorts":[{"ports":[{"port":"6443"}]}]},{"toEndpoints":[{"matchLabels":{"io.kubernetes.pod.namespace":"kube-system","k8s-app":"kube-dns"}}],"toPorts":[{"ports":[{"port":"53","protocol":"UDP"}],"rules":{"dns":[{"matchPattern":"*"}]}}]},{"toPorts":[{"ports":[{"port":"443"}]}]}],"endpointSelector":{},"ingress":[{"fromEndpoints":[{"matchLabels":{"app.kubernetes.io/instance":"prometheus-stack-kube-prom-prometheus","app.kubernetes.io/name":"prometheus","io.kubernetes.pod.namespace":"prometheus-stack"}}],"toPorts":[{"ports":[{"port":"8080","protocol":"TCP"}],"rules":{"http":[{}]}}]},{"fromEntities":["kube-apiserver","remote-node","cluster","world","host"],"toPorts":[{"ports":[{"port":"443","protocol":"TCP"}]},{"ports":[{"port":"10250","protocol":"TCP"}]}]},{"fromEndpoints":[{"matchLabels":{"io.kubernetes.pod.namespace":"kube-system","k8s-app":"konnectivity-agent"}}],"toPorts":[{"ports":[{"port":"10250"}]}]}]}}
  creationTimestamp: "2024-04-22T11:53:12Z"
  generation: 1
  labels:
    argocd.argoproj.io/instance: external-secrets
  name: default
  namespace: external-secrets
  resourceVersion: "53876914"
  uid: a2e4920e-aae1-4fbd-8f70-4c7a3f678eb9
spec:
  egress:
  - toPorts:
    - ports:
      - port: "6443"
  - toEndpoints:
    - matchLabels:
        io.kubernetes.pod.namespace: kube-system
        k8s-app: kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: UDP
      rules:
        dns:
        - matchPattern: '*'
  - toPorts:
    - ports:
      - port: "443"
  endpointSelector: {}
  ingress:
  - fromEndpoints:
    - matchLabels:
        app.kubernetes.io/instance: prometheus-stack-kube-prom-prometheus
        app.kubernetes.io/name: prometheus
        io.kubernetes.pod.namespace: prometheus-stack
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      rules:
        http:
        - {}
  - fromEntities:
    - kube-apiserver
    - remote-node
    - cluster
    - world
    - host
    toPorts:
    - ports:
      - port: "443"
        protocol: TCP
    - ports:
      - port: "10250"
        protocol: TCP
  - fromEndpoints:
    - matchLabels:
        io.kubernetes.pod.namespace: kube-system
        k8s-app: konnectivity-agent
    toPorts:
    - ports:
      - port: "10250"
---
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"cilium.io/v2","kind":"CiliumNetworkPolicy","metadata":{"annotations":{},"labels":{"argocd.argoproj.io/instance":"external-secrets"},"name":"azure","namespace":"external-secrets"},"spec":{"egress":[{"toFQDNs":[{"matchName":"login.microsoftonline.com"},{"matchPattern":"*.vault.azure.net"}],"toPorts":[{"ports":[{"port":"443"}]}]}],"endpointSelector":{"matchLabels":{"app.kubernetes.io/name":"external-secrets"}}}}
  creationTimestamp: "2024-04-22T11:53:12Z"
  generation: 1
  labels:
    argocd.argoproj.io/instance: external-secrets
  name: azure
  namespace: external-secrets
  resourceVersion: "53876913"
  uid: c71387fc-1a09-4ee2-add3-3684d6ad114f
spec:
  egress:
  - toFQDNs:
    - matchName: login.microsoftonline.com
    - matchPattern: '*.vault.azure.net'
    toPorts:
    - ports:
      - port: "443"
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: external-secrets

Cilium Users Document

• Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

• I agree to follow this project's Code of Conduct

[Pionerd]

Issue is no longer present in v1.15.4.