Pods loosing connection to each other on particular node from time to time #70
Comments
|
@theatrus I think this is the same issue I discussed with you on Gitter a while back where it appears the ENI is not forwarding packets to the ipvlan interface correctly for certain pod IPs. Might be due to some bug or race condition with the ipvlan module for our kernel version. Unloading and reloading the module resolves the issue. |
|
disclaimer: I don't work on the lyft cni-ipvlan plugin, but I do use it in production on hundreds of nodes so I've debugged a variety of issues over time.
Note that at the lowest layer, packets are addressed to MAC addresses (be it the mac address of end IP, or the gateway). To discover the MAC address, an ARP broadcast packet is sent out saying "who has this IP?" and an ICMP packet won't be sent until the ARP packet receives a response. More questions:
|
|
@JiriHorky and I work together, so I'll jump in and add some details.
and then on the node: and then on the host: |
|
And now for |
|
|
|
|
|
Thank you @dbyron0 for stepping in and providing the output! One more think I wanted to point out @PaulFurtado, that I probably didn't stress out enough - the strange thing is that the new pod "jhorky..." can not ping the old pod, whereas it works the other way! And what's strange is that when the ping works , I still didn't see any ICMP packets in the tcpdump running on the affected "jhorky..." pod (nor the compute node) so I have no idea wtf was actually replying to those pings... |
Summary of useful stuff I pulled from all those things pasted:
relevant jhorky-shell ipv4 routes: relevant fluentd-loggly-hb9lf ipv4 routes: So jhorky-shell (100.96.254.43) should be dumping the packets destined for 100.96.236.178 directly into eth0 and ipvlan should figure out where they go internally. ARP stuff:Note that all ipvlan slave interfaces share the same MAC address as the ENI so the ARP responses for both IPs should return 0a:b5:1f:03:d1:d6 as the MAC. For the successful ping from fluentd to jhorky, tcpdump inside jhorky shows the correct ARP reply being sent: However, the compute node shows an odd ARP reply: In this case, the host's eth0 IP (10.110.174.111) wants to know where 100.96.236.178 (fluentd) is and the response is Also odd: jhorky's ARP cache is 100% empty. But maybe that's just because it hadn't pinged anything recently. More stuff I'm interested in:
|
|
Hi, I work together /w @JiriHorky @dbyron0
BTW, right now an old(not evicted pods: fluentd, traefik) are able to comunicate with rest of cluster and also to the internet |
This is expected because it is not able to receive ARP responses for pod IPs. The pod is able to communicate with the node IP which shows up in the cache as expected. The affected pod is also able to ping public IPs When attempting to ping another pod IP, that address briefly shows up in the cache as failed, then disappears after a moment. |
|
Out of curiosity, which version of kube-proxy are you using and in which mode? An new ARP sysctl was introduced in 1.13 which broke ARP for us. Issue here: kubernetes/kubernetes#71555 This only applies to a few versions of kube-proxy (it now defaults to the previous behavior) and only in IPVS mode (which you don't seem to be using based on the output of |
|
We are using |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hello,
we are seeing quite strange and (at least for me) hard to debug issue when a Kuberetes node gets into a state where any pod running on it cannot reach any other pods withing the cluster including pods running on the very same node.
We have a node being in this situation right now, so I can provide any debug output if needed (the node is cordened from Kubernetes and all production pods are drained from it).
Let me describe our setup first:
We only have two pods running on the node now:
The pods can't see each other even though they are in the same (/19) subnet and running on the same node:
In a tcpdump (running with -i any) on the compute node, I see these ARP requests but no replies:
When trying to ping the other way around, the situation is very different:
The ping works:
The tcpdump running on the "debug jhorky" docker shows: just this (no ICMP messages?!?):
The tcpdump running on the compute node doesn't show any icmp as well:
Anyway, right now, I have no idea what more to look at.
Once again, the node is in this state, so I can provide any output needed.
Any help much appreciated.
The text was updated successfully, but these errors were encountered: