k8s ingress network policy does not work with L2 announcement

[sittg]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

I have syslog server running in k8s cluster, service is exposed using l2 announcement, I want to control access to the service using network policy.

Syslog is exposed using following service:

apiVersion: v1
kind: Service
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/hostname: syslog.example.com
    external-dns.alpha.kubernetes.io/target: 10.0.46.57
  labels:
    app.kubernetes.io/name: syslog-logger
  name: syslog-logger
  namespace: syslog-infra-logger
spec:
  externalIPs:
    - 10.0.46.57
  internalTrafficPolicy: Cluster
  ipFamilies:
    - IPv4
  ipFamilyPolicy: SingleStack
  ports:
    - name: syslog-tcp    
      port: 514
      protocol: TCP
      targetPort: 514
    - name: syslog-udp
      port: 514
      protocol: UDP
      targetPort: 514
  selector:
    app: axosyslog-collector
  sessionAffinity: None
  type: ClusterIP

We are using default deny np, then one can explicitly allow specific network traffic using additional network policies. It looks like this:

1. default deny network policy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: syslog-infra-logger
spec:
  podSelector: {}
  policyTypes:
  - Ingress

2. allow syslog traffic from lab IPs network policy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-syslog
  namespace: syslog-infra-logger
spec:
  podSelector:
    matchLabels:
      app: axosyslog-collector
  policyTypes:
    - Ingress
  ingress:
    - from:
        - ipBlock:
            cidr: 10.0.40.0/21  # LAB IPs only
      ports:
        - port: 514
          protocol: UDP
        - port: 514
          protocol: TCP

This policy is ignored. During troubleshooting using Hubble UI we noticed, that source IP is the IP of the announced k8s node - cilium_host interface (10.42.5.104), the external source IP of the syslog client is not visible in Hubble at all.
I also tried to add cilium host interface to network policy explicitly (IP/subnet), it was not working either. The only way it was working is to specify 0.0.0.0/0 in the network policy (whitelisting everything)

Hubble screenshot:

Is this correct behavior? Is it possible to control network access using network policies (ingress traffic) when L2 announcement is enabled? What is the recommended way to control ingress traffic in case L2 announcement is enabled?
L2 announcement docs does not contain any related information (did not find anything useful)

Thanks for any update / links to docs or anything related.

Cilium Version

v1.15.1

Kernel Version

Linux worker-e961021e-8n4mg 5.15.0-92-generic #102 SMP Wed Jan 10 09:33:48 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Kubernetes Version

v1.28.8

Regression

No response

Sysdump

No response

Relevant log output

Apr 22 08:50:21.264: 10.42.5.104:51822 (world) -> syslog-infra-logger/axosyslog-collector-89dvt:514 (ID:41252) to-overlay FORWARDED (TCP Flags: SYN)
Apr 22 08:50:21.264: 10.42.5.104:51822 (world) <> syslog-infra-logger/axosyslog-collector-89dvt:514 (ID:41252) policy-verdict:none INGRESS DENIED (TCP Flags: SYN)
Apr 22 08:50:21.264: 10.42.5.104:51822 (world) <> syslog-infra-logger/axosyslog-collector-89dvt:514 (ID:41252) Policy denied DROPPED (TCP Flags: SYN)
Apr 22 08:50:22.295: 10.42.5.104:51822 (world) <> syslog-infra-logger/axosyslog-collector-89dvt:514 (ID:41252) policy-verdict:none INGRESS DENIED (TCP Flags: SYN)
Apr 22 08:50:22.295: 10.42.5.104:51822 (world) -> syslog-infra-logger/axosyslog-collector-89dvt:514 (ID:41252) to-overlay FORWARDED (TCP Flags: SYN)
Apr 22 08:50:22.295: 10.42.5.104:51822 (world) <> syslog-infra-logger/axosyslog-collector-89dvt:514 (ID:41252) Policy denied DROPPED (TCP Flags: SYN)
Apr 22 08:50:24.305: 10.42.5.104:51822 (world) <> syslog-infra-logger/axosyslog-collector-89dvt:514 (ID:41252) policy-verdict:none INGRESS DENIED (TCP Flags: SYN)
Apr 22 08:50:24.305: 10.42.5.104:51822 (world) <> syslog-infra-logger/axosyslog-collector-89dvt:514 (ID:41252) Policy denied DROPPED (TCP Flags: SYN)

Anything else?

No response

Cilium Users Document

• Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

• I agree to follow this project's Code of Conduct

[brb]

Thanks for the issue. Does your request hit the node which runs the syslog pod, or does it get first redirected by an intermediate node?

[sittg]

Hi @brb, I am not sure, what you mean by "redirected by an intermediate node". After another debugging session using Hubble I have added additional Cilium network policy (see below) but the result is still the same.

kind: CiliumNetworkPolicy
metadata:
  annotations:
  name: syslog-in
  namespace: syslog-infra-logger
spec:
  endpointSelector:
    matchLabels:
      app: axosyslog-collector
  ingress:
  - fromEntities:
    - remote-node
    toPorts:
    - ports:
      - port: "514"
        protocol: UDP
      - port: "514"
        protocol: TCP

With network policy 0.0.0.0/0 in place all 10 syslog request will reach syslog server (multiple pods reached - running as daemonset)

When I change the network policy to anything else (e.g. 10.0.0.0/8), I will receive only requests that are routed to the syslog pod running on the announced node

Please note, that bastion (10.0.41.20) is the host, from which I run the logger command to send testing messages. 10.42.6.217 is the IP of cilium host adapter on another worker node. It is strange that this IP does not change even when the request is routed to the different pod!

Do you have any idea, what am I doing wrong? Should I try to use Cilium network policy instead? Thanx!