Loadbalancer IP does not work with cilium

[nitang22]

Hi,
I have a very unique use case here. The loadbalancer IP does not working with Cilium as CNI using kubeProxyReplacement=true as shown in those setup here and here. It works using kube-proxy.

I can't pinpoint exactly the issue but I get a connection reset by peer. Seems like it can reach the ip but cannot reach the envoy proxy in cilium

cloud-provider-kind.log

Tried the same setup on AKS, works fine.

[aojea]

Interesting , have you tried to connect to the nodePort in one of the nodes and see if that works? (30929 in your example)

[aojea]

No answer, so I'll close it, feel free to reopen if you can provide more information

[nitang22]

nodePort works, it comes from the integration between kind and cloud-provider-kind.
Can't reopen the issue on my side, can you do it? @aojea

[aojea]

nodePort works, it comes from the integration between kind and cloud-provider-kind. Can't reopen the issue on my side, can you do it? @aojea

/reopen

absolutely

It is interesting it fails since the packet that arrives to the node comes proxied from the container and is directed to the nodeport ... maybe some cilium config about nodeports that is not liking that flow? but that will be a non-standard behavior most likely

[nitang22]

That's the tricky part, couldn't test on other emulator like minikube since it does not support deactivating the kubeproxy. On Azure, it works. I will try to test on other emulator as well.

[danwinship]

Random guess: Cilium is probably assuming that since the LoadBalancerStatus has an IP rather than a Hostname, that that means it's using VIP-mode load balancing like GCE rather than Proxy-mode load balancing like AWS. Which is to say, Cilium probably hasn't been updated for KEP-1860 yet. (Despite the low KEP number, this is actually a recent feature.)