Hetzner Load Balancer only allow one target if encryption is enabled

[Funzinator]

If I configure a load balancer on TCP mode (not sure about the other modes) and have enable_encryption: true in the cluster config, the load balancer only allows one target (health check fails on all nodes except one):

It only works for nodes where a traefik pod instance is running.
Since it goes away if I disable encryption, I assume it has to do with wireguard.
It works fine, however (with encryption enabled) on public IP (load-balancer.hetzner.cloud/use-private-ip: "false") after allowing the respective node ports to the firewall manually.

For now I will use public IPs instead but would appreciate any pointers.

my traefik helm values:

ports:
  web:
    redirectTo:
      port: websecure
    proxyProtocol:
      insecure: true
  websecure:
    expose: true
    proxyProtocol:
      insecure: true
  metrics:
    expose: false

ingressRoute:
  dashboard:
    enabled: false

tlsOptions:
  default:
    minVersion: VersionTLS12
    cipherSuites:
      - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
      - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
      - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
      - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
      - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
      - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
    sniStrict: true
    preferServerCipherSuites: true

service:
  annotations:
    load-balancer.hetzner.cloud/name: "traefik"
    load-balancer.hetzner.cloud/location: "fsn1"
    load-balancer.hetzner.cloud/type: "lb11"
    load-balancer.hetzner.cloud/protocol: "tcp"
    load-balancer.hetzner.cloud/ipv6-disabled: "false"
    load-balancer.hetzner.cloud/uses-proxyprotocol: "true"
    load-balancer.hetzner.cloud/disable-private-ingress: "false"
    load-balancer.hetzner.cloud/use-private-ip: "true"
    load-balancer.hetzner.cloud/health-check-interval: "10s"
    load-balancer.hetzner.cloud/health-check-timeout: "10s"
    load-balancer.hetzner.cloud/node-selector: "node-role.kubernetes.io/master != true"
  spec:
    externalTrafficPolicy: Local

autoscaling:
  enabled: true
  minReplicas: 2
  maxReplicas: 5

and the relevant sections from cluster_config.yaml:

---
cluster_name: seiyuu
k3s_version: v1.27.7+k3s2

ssh_allowed_networks:
  - 0.0.0.0/0
api_allowed_networks:
  - 0.0.0.0/0

schedule_workloads_on_masters: false

private_network_subnet: 10.21.0.0/16
disable_flannel: false
enable_encryption: true

image: rocky-9
autoscaling_image: rocky-9

cloud_controller_manager_manifest_url: "https://github.com/hetznercloud/hcloud-cloud-controller-manager/releases/download/v1.18.0/ccm-networks.yaml"
csi_driver_manifest_url: "https://raw.githubusercontent.com/hetznercloud/csi-driver/v2.5.1/deploy/kubernetes/hcloud-csi.yml"
system_upgrade_controller_manifest_url: "https://raw.githubusercontent.com/rancher/system-upgrade-controller/master/manifests/system-upgrade-controller.yaml"

masters_pool:
  instance_type: cax11
  instance_count: 1
  location: fsn1

worker_node_pools:
- name: cax11-autoscale-1
  instance_type: cax11
  instance_count: 2
  location: fsn1

[vitobotta]

Interesting. I am not using encryption myself since I am fine without since it's using a private network (I don't think there is a big risk), so I haven't noticed this before. Have you tried with another ingress controller to see if it's a traefik thing just in case? Might be that, a bug with the Hetzner cloud controller manager or k3s support for wireguard. No idea. I suggest you check first if it happens with e.g. ingress nginx as well, if not I'd open an issue on the Hetzner CCM repo. I am converting this to a discussion since it doesn't seem to be a problem specific to hetzner-k3s.

[vitobotta]

I kinda got used to nginx over time. It just works, it's fast, lots of features and and well documented

[Funzinator]

At the end of the day, these are all just reverse proxies. 😄

[vitobotta]

True. I just use nginx out of habit now, haven't tried others in quite a while

[Funzinator]

Then give Traefik another chance maybe 😊

[vitobotta]

I could :)