New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
kube-proxy ipvs mode hangs after a few hours, needs manual restart #71071
Comments
/sig network |
the same error. upgrade from 1.11.1. Everything become good except kube-proxy becomse not stable, some time pod can't access svc though their endpoints are ok .. And the same as the name resolution. so i think it may exist an important bug here. |
/area ipvs |
@gjcarneiro do you resolve this problem? |
Well, I switched the cluster back to the default (iptables) mode, appears to be pretty stable since then... |
@kubernetes/sig-network-bugs |
@berlinsaint: Reiterating the mentions to trigger a notification: In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
same problem ,both 1.12.2 and 1.13.0-beta1. @m1093782566 @Lion-Wei |
@gjcarneiro do you mean that in kernel 4.15 , it has the same problem? |
你邮箱多少啊啊,,认识下。。少年 |
点我头像就看到了 |
I am also seeing this repeatedly in my centralized logging system:
I'm running v1.12.3 (fresh install, not an upgrade). |
Hi, all. Thanks for the report, I'll make a research and try to figure out what happened, any progress I'll report here. |
Besides what happened, which is good to find out, I don't feel confident about IPVS until there is code in place that detects and resets a broken netlink socket. Because from what I can guess from the logs and traces it is that the netlink socket gets into a broken state and the task that reads/writes from/to it will hang forever. I think kube-proxy needs to have code that detects this problem, and if it happens create a new netlink socket from scratch, and maybe restart the task that handles it. But, no, I'm not volunteering, I don't even know Go, sorry. |
We upgraded the go netlink version some days ago for supporting graceful termination. The netlink library was running pretty stable before. I am not sure if it's the cause. |
1.12.3 with the same error. E1129 01:01:10.862180 1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
E1129 21:09:11.422553 1 proxier.go:1485] Failed to get IPVS service, error: Expected only one service obtained=0
E1129 21:09:11.422699 1 proxier.go:1116] Failed to sync endpoint for service: 172.17.0.1:31800/TCP, err: Expected only one service obtained=0
E1129 01:01:11.618125 1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
E1129 01:01:11.905626 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:01:41.771596 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:02:11.923872 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:02:42.060658 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:03:12.326762 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:03:42.718417 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:04:12.859933 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:04:42.963527 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:05:13.078486 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:05:43.185141 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:06:13.303090 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:06:43.408582 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:07:13.550358 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:07:43.686235 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:08:13.786326 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:08:43.981764 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:09:14.188053 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:09:44.316316 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:10:14.440466 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:10:44.550092 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:11:14.658190 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:11:44.765396 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:12:14.975724 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:12:45.085632 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:13:15.317679 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:13:45.432469 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:14:15.689681 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:14:45.813565 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:15:33.434931 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:16:03.578017 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:16:33.709744 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:17:03.823325 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:17:33.935191 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:18:04.053183 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:18:34.248994 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:19:04.375946 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:19:34.552779 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:19:36.596182 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:19:47.433846 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:20:17.534206 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:20:47.682350 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:21:17.825175 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:21:47.942548 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:22:18.055820 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:22:48.149494 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:23:18.564985 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:23:48.678462 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:24:18.836217 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:24:48.991344 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:25:19.091032 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:25:49.187178 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:26:19.301770 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:26:49.425812 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:27:19.606711 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:27:49.721224 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:28:20.109117 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:28:50.211574 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:29:20.405007 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:29:50.507766 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:30:20.605471 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:30:50.702798 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:31:20.807400 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:31:50.903049 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:32:21.005618 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:32:51.106378 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:33:21.224447 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:33:51.354916 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:34:21.467898 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:34:51.583216 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:35:21.691099 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:35:51.797151 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:36:21.921057 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:36:52.083821 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:37:02.445737 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:37:07.427811 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:37:37.546362 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:38:07.651769 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:38:37.797858 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:39:08.064112 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:39:38.181089 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:40:08.282892 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:40:38.390167 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:41:08.508549 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:41:38.617921 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:42:08.728384 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:42:38.844074 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:43:08.973517 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:43:39.092095 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:44:09.202329 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:44:39.363721 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:45:09.476240 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:45:39.764944 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:46:09.894587 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:46:40.013388 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:47:10.127894 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:47:40.236119 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:48:10.397111 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:48:40.600084 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:49:10.877034 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:49:41.023593 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:50:11.149894 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:50:41.262320 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:51:11.410407 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:51:41.547057 1 proxier.go:1622] Failed to unbind service addr fe80::1c5d:70ff:fef2:e97b from dummy interface kube-ipvs0: error unbind address: fe80::1c5d:70ff:fef2:e97b from interface: kube-ipvs0, err: cannot assign requested address
E1129 01:52:11.609974 1 proxier.go:1544] Failed to add destination: 172.31.200.9:443, error: file exists
E1129 01:52:11.610088 1 graceful_termination.go:89] Try delete rs "10.96.0.1:443/TCP/172.31.10.38:6443" err: invalid argument
E1129 01:52:11.616690 1 proxier.go:1485] Failed to get IPVS service, error: Expected only one service obtained=0
E1129 01:52:11.616767 1 proxier.go:1116] Failed to sync endpoint for service: 172.18.0.1:31645/TCP, err: Expected only one service obtained=0
E1129 01:01:12.181112 1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
E1129 01:01:22.278474 1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
E1129 01:01:21.459378 1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
E1129 01:01:17.442316 1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
E1129 20:10:17.955872 1 proxier.go:1455] Failed to add IPVS service "dev-fffrf/mysql-testportal:mysql": file exists
E1129 20:10:17.955863 1 graceful_termination.go:89] Try delete rs "10.100.217.68:15004/TCP/192.100.143.225:15004" err: Failed to delete rs "10.100.217.68:15004/TCP/192.100.143.225:15004", can't find the real server
E1129 20:10:17.955892 1 proxier.go:1119] Failed to sync service: 172.17.0.1:31645/TCP, err: file exists
E1129 20:10:17.956412 1 proxier.go:1485] Failed to get IPVS service, error: invalid argument
E1129 20:10:17.956417 1 graceful_termination.go:89] Try delete rs "10.96.0.1:443/TCP/172.31.10.38:6443" err: Failed to delete rs "10.96.0.1:443/TCP/172.31.10.38:6443", can't find the real server
E1129 20:10:17.956426 1 proxier.go:1116] Failed to sync endpoint for service: 172.31.200.67:31645/TCP, err: invalid argument
E1129 20:10:17.956448 1 graceful_termination.go:89] Try delete rs "10.101.1.98:5672/TCP/192.100.29.180:5672" err: invalid argument
E1129 20:10:17.956547 1 graceful_termination.go:183] Try flush graceful termination list err
E1129 01:01:17.484679 1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
E1129 02:52:17.498135 1 graceful_termination.go:89] Try delete rs "10.96.0.1:443/TCP/172.31.10.38:6443" err: invalid argument
E1129 02:52:17.498396 1 graceful_termination.go:183] Try flush graceful termination list err
E1129 06:28:17.607394 1 graceful_termination.go:89] Try delete rs "10.96.0.1:443/TCP/172.31.10.38:6443" err: device or resource busy
E1129 06:28:17.607513 1 graceful_termination.go:183] Try flush graceful termination list err
E1129 10:02:17.718458 1 proxier.go:1544] Failed to add destination: 192.100.183.139:3306, error: file exists
E1129 10:02:17.718606 1 proxier.go:1544] Failed to add destination: 192.100.183.152:8080, error: invalid argument```
|
BTW. from above my logs , kube-proxy seems like still have many problems with iptables-restore . all my node are UBUNTU 16.04 which have iptable's version 1.6.0. |
We see this happening when upgrading from 1.12.1. it must thus be in the delta of 1.12.1 and 1.12.2. |
The issue is graceful termination related and I think we have fixed the bug in v1.13.1. |
@m1093782566 will the fix be backported to the 1.12 line? right now this is blocking the 1.12.3 security fix for us. |
Got the same with upgrade from 1.11.2 to 1.11.5. |
@m1093782566 can you provide more information on the fix? I would be interested to look over the commit. |
This is pretty insane, we got bitten really hard earlier today. |
Was the bug chery-picked? |
any chance for a new build in v1.11.x ? |
I've been refreshing the releases pages pretty much every hour for the last couple of days in the hope 1.12.5 will appear so i can patch this bug on multiple clusters. :-) |
@verwilst haha, same, can't wait for v1.11.7 to get released to upgrade current prod clusters. And v1.12.5 too to start upgrading in lower envs. |
Its here :-P |
Why is it not there for 1.11? :( |
Was this bug fixed in 1.11.7? I didn't see any release notes. |
Yes it is https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.11.md#v1117:
|
Yes I forgot to add "IPVS" in the release-note in the PR, sorry about that |
We've had no issues since upgrading to 1.12.5 🎉 |
Great news! |
@m1093782566 , @thockin I think we can close this issue (no new reports in about 2 months) For reference, bug impacts initial releases with graceful termination:
Fix is present after:
|
/close |
@m1093782566: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Can we reopen? We're seeing this issue on 1.12.7. |
yes, we are seeing the issues in 1.12.9 too |
Are you sure it is the same issue? |
sorry, it may have been a false alert... We see this is the logs and its not the same error
for some reason, when the cluster is under load it couldn't route to some of the cluster service IPs. We are still not sure if it is an IPVS issue. |
Maybe the endpoint propagation is taking time if the apiserver is overloaded? The messages are normal Info level messages: https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/graceful_termination.go#L161 |
I can confirm the same log in 1.14.4. When does this log come out?
|
These are just information logs:
@andrewsykim Maybe we could now decrease the level of these logs so they only show above v=5 for instance ? |
What happened: upgraded from v1.11.0 to v1.12.2, now kube-proxy hangs, usually in less than a day
What you expected to happen: it should continue working
How to reproduce it (as minimally and precisely as possible): just have a normal kube-proxy running in ipvs mode. It gets stuck usually in less than a day.
Anything else we need to know?:
I've sent SIGABRT to the kube-proxy process to get a stack trace, and the only thing interesting is that when the bug happens goroutine 1 is reading from netlink, whereas when I send SIGABRT and the process is not stuck it is not reading from netlink.
The last lines that it logged were:
If I restart kube-proxy (manually, ugh), it all recovers and works fine. But it will get stuck again, eventually.
Environment:
kubectl version
): v1.12.2uname -a
): 4.15.0-39-generic (but get it on the original 4.4 kernel as well)/kind bug
The text was updated successfully, but these errors were encountered: