kube-proxy in ipvs mode does not synchronize ipvs status correctly

[randmonkey]

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug
/sig network
/area kube-proxy

/kind feature

What happened:
We run a kubernetes cluster using k8s 1.8.0 and kube-proxy running in ipvs mode. Services created/updated after kube-proxy(pod) started running are not synchronized in ipvs, but services deleted before are still being synced.
What you expected to happen:
The new services are synced to ipvs, but the old ones are not synced.
How to reproduce it (as minimally and precisely as possible):
0. run k8s 1.8.0 cluster with kube-proxy run in ipvs mode

1. create a service svc-1 and get its clustetIP
2. run ipvsadm -ln, we cannot see records for the clusterIP after a very long time(>5min),
3. restart kube-proxy, and we can see the IP in ipvsadm -ln.

Anything else we need to know?:
args for running kube-proxy:

      containers:
      - command:
        - /usr/local/bin/kube-proxy
        - --kubeconfig=/var/lib/kube-proxy/kubeconfig.conf
        - --cluster-cidr=10.0.0.0/10
        - --masquerade-all=true
        - --feature-gates=SupportIPVSProxyMode=true
        - --proxy-mode=ipvs
        - --ipvs-min-sync-period=5s
        - --ipvs-sync-period=20s
        - --ipvs-scheduler=rr
        - --v=3

logs of kube-proxy(pod):

I1116 04:01:28.004725       1 proxier.go:385] Updating existing service "svc-test/svc-12:http" at 172.16.154.181:80/TCP
I1116 04:01:28.004905       1 proxier.go:863] Syncing ipvs Proxier rules
I1116 04:01:28.013591       1 proxier.go:1253] IPVS service svc-test/svc-1:http was changed
I1116 04:01:28.024978       1 proxier.go:1253] IPVS service svc-test/nginx-2:http was changed
I1116 04:01:28.035614       1 proxier.go:1253] IPVS service svc-test/svc-9:http was changed
I1116 04:01:28.046024       1 proxier.go:1245] Adding new service "svc-test/svc-12:http" 172.16.154.181:80/TCP
I1116 04:01:28.055985       1 proxier.go:1253] IPVS service kube-system/kube-dns:dns was changed
I1116 04:01:28.065551       1 proxier.go:1253] IPVS service net-perf-test/netperf-server:netperf-control was changed
I1116 04:01:28.074465       1 proxier.go:1253] IPVS service svc-test/svc-4:http was changed
I1116 04:01:28.083380       1 proxier.go:1253] IPVS service default/kubernetes:https was changed
I1116 04:01:28.091991       1 proxier.go:1253] IPVS service svc-test/svc-7:http was changed
I1116 04:01:28.100231       1 proxier.go:1253] IPVS service kube-system/kube-dns:dns-tcp was changed
I1116 04:01:28.108334       1 proxier.go:1253] IPVS service net-perf-test/netperf-server:netperf-data-tcp was changed
I1116 04:01:28.117312       1 proxier.go:1253] IPVS service svc-test/svc-2:http was changed
I1116 04:01:28.125229       1 proxier.go:1253] IPVS service svc-test/svc-11:http was changed
I1116 04:01:28.132750       1 proxier.go:1253] IPVS service svc-test/svc-8:http was changed
I1116 04:01:28.140141       1 proxier.go:1253] IPVS service kube-system/nuke-controller: was changed
I1116 04:01:28.151349       1 proxier.go:1253] IPVS service kube-system/nuke-controller: was changed
I1116 04:01:28.151486       1 proxier.go:1253] IPVS service kube-system/nuke-controller: was changed
I1116 04:01:28.151585       1 proxier.go:1253] IPVS service svc-test/svc-6:http was changed
I1116 04:01:28.158658       1 proxier.go:1253] IPVS service svc-test/svc-3:http was changed
I1116 04:01:28.166020       1 proxier.go:1253] IPVS service svc-test/svc-5:http was changed
I1116 04:01:28.173177       1 proxier.go:1253] IPVS service svc-test/svc-10:http was changed

note: services svc-test/svc-* are deleted.

currently active services & endpoints:

# kubectl get svc --all-namespaces
NAMESPACE       NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)               AGE
default         kubernetes        ClusterIP   172.16.0.1      <none>        443/TCP               8d
default         nginx-1           ClusterIP   172.16.1.1      <none>        80/TCP                3h
kube-system     kube-dns          ClusterIP   172.16.0.10     <none>        53/UDP,53/TCP         8d
kube-system     nuke-controller   NodePort    172.16.93.129   <none>        9500:30001/TCP        8d
net-perf-test   netperf-server    ClusterIP   172.16.56.26    <none>        12865/TCP,56821/TCP   18h
svc-test        nginx-2           ClusterIP   172.16.2.2      <none>        80/TCP                18h

# kubectl get ep --all-namespaces
NAMESPACE       NAME                       ENDPOINTS                                               AGE
default         kubernetes                 10.200.20.75:6443,10.200.20.76:6443,10.200.20.77:6443   8d
default         nginx-1                    10.23.0.15:80                                           3h
kube-system     cloud-controller-manager   <none>                                                  8d
kube-system     kube-controller-manager    <none>                                                  8d
kube-system     kube-dns                   10.200.20.79:53,10.200.20.79:53                         8d
kube-system     kube-scheduler             <none>                                                  8d
kube-system     nuke-controller            10.200.20.75:9500                                       8d
net-perf-test   netperf-server             10.29.0.4:56821,10.29.0.4:12865                         18h
svc-test        nginx-2                    10.45.0.6:80                                            18h

note: service default/nginx-1 is created after last restart of kube-proxy on this node.

output of ipvsadm -ln:

# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.16.0.1:443 rr persistent 10800
  -> 10.200.20.75:6443            Masq    1      0          0         
  -> 10.200.20.76:6443            Masq    1      0          0         
  -> 10.200.20.77:6443            Masq    1      1          0         
TCP  172.16.0.10:53 rr
  -> 10.200.20.79:53              Masq    1      0          0         
TCP  172.16.1.1:80 rr
TCP  172.16.2.2:80 rr
  -> 10.45.0.6:80                 Masq    1      0          0         
TCP  172.16.56.26:12865 rr
  -> 10.29.0.4:12865              Masq    1      0          0         
TCP  172.16.56.26:56821 rr
  -> 10.29.0.4:56821              Masq    1      0          0         
TCP  172.16.93.129:9500 rr
  -> 10.200.20.75:9500            Masq    1      0          0         
TCP  10.200.20.75:30001 rr
  -> 10.200.20.75:9500            Masq    1      0          0         
TCP  10.200.30.200:30001 rr
  -> 10.200.20.75:9500            Masq    1      0          0         
UDP  172.16.0.10:53 rr
  -> 10.200.20.79:53              Masq    1      0          0 

Environment:

• Kubernetes version (use kubectl version): 1.8.0
• Cloud provider or hardware configuration: No cloud provider
• OS (e.g. from /etc/os-release): Ubuntu 16.04 LTS
• Kernel (e.g. uname -a): 4.4.0-72-generic x64
• Install tools: kubeadm
• Others:

[randmonkey]

@m1093782566

[m1093782566]

Thanks for reporting it.

/assign

[m1093782566]

Will take a deep look...

[m1093782566]

I WOULD suggest you check the head of master branch. There is a known issue in v1.8.0.

Seems your issue if very similar to #52393

[randmonkey]

Is there a released version that solves #52393 and possibly solve this?

[m1093782566]

I would suggest you try ipvs proxy in v1.9 since it still has some known issues now - we target beta in v1.9.

[m1093782566]

/area ipvs

[m1093782566]

I believe it's fixed in v1.9

I am going to close this issue now. Please re-open if it still persist.

[m1093782566]

/close

[miry]

@m1093782566 I have similar problem for kubernetes v1.11.3 Do you know what the best way to debug it?

My situation:

On one node I see the service and pod interfaces:

1.1.1.1# ip addr show | grep  10.244.254
inet 10.244.254.0/32 scope global flannel.1
1.1.1.1# ip addr show | grep  10.101.80.23
inet 10.101.80.23/32 brd 10.101.80.23 scope global kube-ipvs0

and in same time I don't get it from another node neither service or pod.

2.2.2.2#  ip addr show | grep  10.244.254
#

2.2.2.2# ipvsadm -ln
...
TCP  10.101.80.23:8080 rr
  -> 10.244.254.12:8080           Masq    1      0          0 

kube-proxy logs for 2.2.2.2:

E1017 10:40:18.624856       1 proxier.go:423] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
)

How can I debug the issue?

[AndySkyL]

have similar problem for kubernetes v1.12.2 Do you know what the best way to debug it?