ingress/gateway-api: expose listeners on host network

[mhofstetter]

This PR introduces support for exposing the Envoy Listeners created by the Ingress Controller and/or Gateway API directly on the host network. This is useful in some edge cases where one don't want / can't to use K8s LoadBalancer/NodePort functionality (dev environments, cluster-external loadbalancer, ...)

Expose Envoy listeners on host network

This commit adds support for exposing the L7 Envoy Listeners directly on the host network - and no longer use Kubernetes Services of type LoadBalancer or NodePort.

The listener is exposed on all interfaces (0.0.0.0 for IPv4 and/or :: for IPv6).

Enable HostNetwork support via Helm

• Ingress Controller: ingressController.hostNetwork.enabled=true
• Gateway API: gatewayAPI.hostNetwork.enabled=true

Configure listener port

• Shared Ingress: configurable via Helm
  (ingressController.hostNetwork.sharedHTTPPort & ingressController.hostNetwork.sharedTLSPassthroughPort)
• Dedicated Ingress: configurable via Annotation on the resource Ingress
  (ingress.cilium.io/http-host-port & ingress.cilium.io/tls-passthrough-host-port)
• Gateway API: configurable via spec.listeners.port on the resource Gateway

Be aware that missconfiguration might result in port clashes.

Expose Envoy listeners on subset of nodes

This commit adds support for exposing L7 Envoy Listeners only on a subset of Cilium Nodes. This only works in combination with the hostnetwork mode.

Configure node labelselector via Helm

• Ingress Controller: ingressController.hostNetwork.nodes.matchLabels
• Gateway API: gatewayAPI.hostNetwork.nodes.matchLabels

ingressController:
  hostNetwork:
    nodes:
      matchLabels:
        role: infra
        component: ingress

An empty selector selects all Nodes and continues to expose the functionality on all Cilium Nodes.

---

Please review the individual commits.

Fixes: #21390

[mhofstetter]

/test

[gandro]

Helm LGTM!

[mhofstetter]

rebased to main to resolve conflicts

[learnitall]

Docs look good, thanks!

[mhofstetter]

rebased to main and resolved conflicts

[mhofstetter]

/test

[maxpain]

The listener is exposed on all interfaces (0.0.0.0 for IPv4 and/or :: for IPv6).

[2024-03-07 15:37:30.207][7][error][config] [external/envoy/source/extensions/listener_managers/listener_manager/listener_manager_impl.cc:1179] listener 'kube-system/cilium-ingress/listener' failed to bind or apply socket options: cannot bind '0.0.0.0:80': Permission denied
[2024-03-07 15:37:30.210][7][warning][config] [external/envoy/source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:138] gRPC config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) kube-system/cilium-ingress/listener: cannot bind '0.0.0.0:80': Permission denied

[mhofstetter]

The listener is exposed on all interfaces (0.0.0.0 for IPv4 and/or :: for IPv6).

[2024-03-07 15:37:30.207][7][error][config] [external/envoy/source/extensions/listener_managers/listener_manager/listener_manager_impl.cc:1179] listener 'kube-system/cilium-ingress/listener' failed to bind or apply socket options: cannot bind '0.0.0.0:80': Permission denied
[2024-03-07 15:37:30.210][7][warning][config] [external/envoy/source/extensions/config_subscription/grpc/grpc_subscription_impl.cc:138] gRPC config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) kube-system/cilium-ingress/listener: cannot bind '0.0.0.0:80': Permission denied

👋 @maxpain

Thanks for testing the feature. Envoy should have permissions to bind to privileged ports - but that might depend on the actual environment. At least it was working on a local dev kind cluster when i tested this the last time.

If this is not working there's still the option to change the default port for the shared listener (HTTP/HTTPS & TLS Passthrough) to an unprivileged port via the helm values ingressController.hostNetwork.sharedHTTPPort & ingressController.hostNetwork.sharedTLSPassthroughPort.

Otherwise, feel free to raise a GH issue and provide some more information by attaching a sysdump. Thanks!

[mhofstetter]

Update: It looks like binding to privileged ports works on kind clusters because Docker lowers the unprivileged port start from 1024 to 0 (moby/moby#41030)

root@kind-worker:/home/cilium# sysctl net | grep port_start
net.ipv4.ip_unprivileged_port_start = 0

Unfortunately it looks like it's not enough to just add the capability NET_BIND_SERVICE to the Pod where Cilium's Envoy instance is running (either Cilium Agent (embedded mode) or Cilium Proxy (daemonset mode)) - because since Cilium 1.15 the Envoy process itself no longer has the privileges (capabilities are dropped before starting the process) - see #27498.

Therefore, binding to privileged ports will not work in clusters where this sysctl option is not set.

I try to find a solution. In the meantime it's recommended to test the feature by explicitly configuring an unprivileged port.

/cc @maxpain

[kahirokunn]

Amazing!! Thx!!!

[mhofstetter]

updates / related changes:

• envoy: add support to bind to privileged ports #32158: adds support for using privileged ports
• ingress: change hostnetwork default port to unprivileged 8080 #32159: changes the default port from privileged port 80 to 8080 (as privileged ports aren't configured by default)
• Ingress/Gateway API: merge Envoy listeners for HTTP(S) and TLS passthrough #31646: merged HTTP/HTTPS and TLS passthrough Envoy listeners into one. As a result of this, the Helm values (Shared Ingress) and Annnotations (Dedicated Ingress) have changed (as it's no longer necessary to define a port for each of these Listeners) See commit 4cb67f9

Further information can be found in the PRs. Please open a new issue if you have any questions / problems.

/cc @maxpain - and thanks again for reporting!