controller-install sctipt stuck at Waiting for Kubernetes API...

[amit2277]

Hello I am new to kubernetes and want to install test environment with two controller and three worker nodes on 5 physical HP Prolient servers.

I have installed coreos on baremetal server and run controller-install.sh script but it stuck at waiting for kubernetes API (logs at below) -- it seems it some issue with my /etc/resolve.conf but i don't know where and what to change i have tried to change nameserver to 127.0.0.1 but wont succeed. currently it is 8.8.8.8 , Can anyone please guide me.

My current setup :
#cat /etc/resolv.conf.
nameserver 8.8.8.8

cat /etc/hosts

127.0.0.1 localhost
::1 localhost
192.168.51.81 controller

systemctl status etcd2

● etcd2.service - etcd2
Loaded: loaded (/usr/lib/systemd/system/etcd2.service; disabled; vendor preset: disabled)
Drop-In: /run/systemd/system/etcd2.service.d
└─20-cloudinit.conf
Active: active (running) since Fri 2017-07-14 08:48:38 UTC; 1h 27min ago
Main PID: 990 (etcd2)
Tasks: 23 (limit: 32768)
Memory: 64.2M
CPU: 19.540s
CGroup: /system.slice/etcd2.service
└─990 /usr/bin/etcd2

Jul 14 08:48:38 controller etcd2[990]: ce2a822cea30bfca is starting a new election at term 10
Jul 14 08:48:38 controller etcd2[990]: ce2a822cea30bfca became candidate at term 11
Jul 14 08:48:38 controller etcd2[990]: ce2a822cea30bfca received vote from ce2a822cea30bfca at term 11
Jul 14 08:48:38 controller etcd2[990]: ce2a822cea30bfca became leader at term 11
Jul 14 08:48:38 controller etcd2[990]: raft.node: ce2a822cea30bfca elected leader ce2a822cea30bfca at term 11
Jul 14 08:48:38 controller etcd2[990]: published {Name:controller ClientURLs:[http://0.0.0.0:4001 http://192.168.51.81:2379]} to cluster 7e27652122e8b2ae
Jul 14 09:29:01 controller etcd2[990]: start to snapshot (applied: 230023, lastsnap: 220022)
Jul 14 09:29:01 controller etcd2[990]: saved snapshot at index 230023
Jul 14 09:29:01 controller etcd2[990]: compacted raft log at 225023
Jul 14 09:29:08 controller etcd2[990]: purged file /var/lib/etcd2/member/snap/0000000000000009-000000000002bf32.snap successfully

systemctl status flanneld

● flanneld.service - flannel - Network fabric for containers (System Application Container)
Loaded: loaded (/usr/lib/systemd/system/flanneld.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/flanneld.service.d
└─40-ExecStartPre-symlink.conf.conf, 50-network-config.conf
Active: active (running) since Fri 2017-07-14 08:48:41 UTC; 1h 28min ago
Docs: https://github.com/coreos/flannel
Main PID: 1068 (flanneld)
Tasks: 22 (limit: 32768)
Memory: 126.6M
CPU: 1.489s
CGroup: /system.slice/flanneld.service
└─1068 /opt/bin/flanneld --ip-masq=true

Jul 14 08:48:41 controller flannel-wrapper[1068]: I0714 08:48:41.610498 1068 manager.go:124] Searching for interface using 192.168.51.81
Jul 14 08:48:41 controller flannel-wrapper[1068]: I0714 08:48:41.610840 1068 manager.go:149] Using interface with name enp2s0f0 and address 192.168.51.81
Jul 14 08:48:41 controller flannel-wrapper[1068]: I0714 08:48:41.610854 1068 manager.go:166] Defaulting external address to interface address (192.168.51.81)
Jul 14 08:48:41 controller flannel-wrapper[1068]: I0714 08:48:41.661383 1068 local_manager.go:134] Found lease (10.10.81.0/24) for current IP (192.168.51.81), reusing
Jul 14 08:48:41 controller flannel-wrapper[1068]: I0714 08:48:41.720174 1068 ipmasq.go:47] Adding iptables rule: -s 10.10.0.0/16 -d 10.10.0.0/16 -j RETURN
Jul 14 08:48:41 controller flannel-wrapper[1068]: I0714 08:48:41.864413 1068 ipmasq.go:47] Adding iptables rule: -s 10.10.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
Jul 14 08:48:41 controller flannel-wrapper[1068]: I0714 08:48:41.902481 1068 ipmasq.go:47] Adding iptables rule: ! -s 10.10.0.0/16 -d 10.10.0.0/16 -j MASQUERADE
Jul 14 08:48:41 controller flannel-wrapper[1068]: I0714 08:48:41.905284 1068 manager.go:250] Lease acquired: 10.10.81.0/24
Jul 14 08:48:41 controller flannel-wrapper[1068]: I0714 08:48:41.905587 1068 network.go:98] Watching for new subnet leases
Jul 14 08:48:41 controller systemd[1]: Started flannel - Network fabric for containers (System Application Container).

systemctl status fleet
● fleet.service - fleet daemon
Loaded: loaded (/usr/lib/systemd/system/fleet.service; disabled; vendor preset: disabled)
Drop-In: /run/systemd/system/fleet.service.d
└─20-cloudinit.conf
Active: active (running) since Fri 2017-07-14 08:48:38 UTC; 1h 29min ago
Main PID: 1036 (fleetd)
Tasks: 7 (limit: 32768)
Memory: 36.5M
CPU: 1min 32.825s
CGroup: /system.slice/fleet.service
└─1036 /usr/bin/fleetd

Jul 14 08:48:38 controller systemd[1]: Started fleet daemon.
Jul 14 08:48:39 controller fleetd[1036]: INFO fleetd.go:64: Starting fleetd version 0.11.8
Jul 14 08:48:39 controller fleetd[1036]: INFO fleetd.go:170: No provided or default config file found - proceeding without
Jul 14 08:48:39 controller fleetd[1036]: INFO server.go:160: Establishing etcd connectivity
Jul 14 08:48:39 controller fleetd[1036]: INFO server.go:171: Starting server components
Jul 14 08:48:39 controller fleetd[1036]: INFO engine.go:185: Engine leadership acquired

fleetctl list-machines
MACHINE IP METADATA
a682ae32... 192.168.51.81 role=master

etcdctl cluster-health
member ce2a822cea30bfca is healthy: got healthy result from http://0.0.0.0:4001
cluster is healthy

netstat -ntpul
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 192.168.51.81:7001 0.0.0.0:* LISTEN 990/etcd2
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 1248/kubelet
tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN 1687/hyperkube
tcp 0 0 192.168.51.81:2380 0.0.0.0:* LISTEN 990/etcd2
tcp6 0 0 :::22 :::* LISTEN 1/systemd
tcp6 0 0 :::4001 :::* LISTEN 990/etcd2
tcp6 0 0 :::4194 :::* LISTEN 1248/kubelet
tcp6 0 0 :::2375 :::* LISTEN 1/systemd
tcp6 0 0 :::10250 :::* LISTEN 1248/kubelet
tcp6 0 0 :::10251 :::* LISTEN 1610/hyperkube
tcp6 0 0 :::2379 :::* LISTEN 990/etcd2
tcp6 0 0 :::10252 :::* LISTEN 1720/hyperkube
tcp6 0 0 :::10255 :::* LISTEN 1248/kubelet
udp 0 0 192.168.51.81:8285 0.0.0.0:* 1068/flanneld
udp6 0 0 fe80::6ab5:99ff:fec:546 :::* 9020/systemd-network

Can anyone help me where i am doing mistake.

Error:
systemctl status kubelet -l
● kubelet.service
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2017-07-14 08:48:48 UTC; 1h 14min ago
Main PID: 1248 (kubelet)
Tasks: 28 (limit: 32768)
Memory: 768.5M
CPU: 2min 14.584s
CGroup: /system.slice/kubelet.service
├─1248 /kubelet --api-servers=http://127.0.0.1:8080 --register-schedulable=false --cni-conf-dir=/etc/kubernetes/cni/net.d --network-plugin=cni --container-runtime=docker --rkt-path=/usr/bin/rkt --rkt-stage1-image=coreos.com/rk
└─1433 journalctl -k -f

Jul 14 10:02:54 controller kubelet-wrapper[1248]: E0714 10:02:54.625731 1248 reflector.go:188] pkg/kubelet/kubelet.go:386: Failed to list *api.Node: Get http://127.0.0.1:8080/api/v1/nodes?fieldSelector=metadata.name%3D192.168.51.81&re
Jul 14 10:02:55 controller kubelet-wrapper[1248]: E0714 10:02:55.377911 1248 reflector.go:188] pkg/kubelet/kubelet.go:378: Failed to list *api.Service: Get http://127.0.0.1:8080/api/v1/services?resourceVersion=0: dial tcp 127.0.0.1:80
Jul 14 10:02:55 controller kubelet-wrapper[1248]: E0714 10:02:55.378592 1248 reflector.go:188] pkg/kubelet/config/apiserver.go:44: Failed to list *api.Pod: Get http://127.0.0.1:8080/api/v1/pods?fieldSelector=spec.nodeName%3D192.168.51
Jul 14 10:02:55 controller kubelet-wrapper[1248]: E0714 10:02:55.626339 1248 reflector.go:188] pkg/kubelet/kubelet.go:386: Failed to list *api.Node: Get http://127.0.0.1:8080/api/v1/nodes?fieldSelector=metadata.name%3D192.168.51.81&re
Jul 14 10:02:56 controller kubelet-wrapper[1248]: E0714 10:02:56.378822 1248 reflector.go:188] pkg/kubelet/kubelet.go:378: Failed to list *api.Service: Get http://127.0.0.1:8080/api/v1/services?resourceVersion=0: dial tcp 127.0.0.1:80
Jul 14 10:02:56 controller kubelet-wrapper[1248]: E0714 10:02:56.379795 1248 reflector.go:188] pkg/kubelet/config/apiserver.go:44: Failed to list *api.Pod: Get http://127.0.0.1:8080/api/v1/pods?fieldSelector=spec.nodeName%3D192.168.51
Jul 14 10:02:56 controller kubelet-wrapper[1248]: E0714 10:02:56.627007 1248 reflector.go:188] pkg/kubelet/kubelet.go:386: Failed to list *api.Node: Get http://127.0.0.1:8080/api/v1/nodes?fieldSelector=metadata.name%3D192.168.51.81&re
Jul 14 10:02:57 controller kubelet-wrapper[1248]: E0714 10:02:57.379581 1248 reflector.go:188] pkg/kubelet/kubelet.go:378: Failed to list *api.Service: Get http://127.0.0.1:8080/api/v1/services?resourceVersion=0: dial tcp 127.0.0.1:80
Jul 14 10:02:57 controller kubelet-wrapper[1248]: E0714 10:02:57.380463 1248 reflector.go:188] pkg/kubelet/config/apiserver.go:44: Failed to list *api.Pod: Get http://127.0.0.1:8080/api/v1/pods?fieldSelector=spec.nodeName%3D192.168.51
Jul 14 10:02:57 controller kubelet-wrapper[1248]: E0714 10:02:57.627616 1248 reflector.go:188] pkg/kubelet/kubelet.go:386: Failed to list *api.Node: Get http://127.0.0.1:8080/api/v1/nodes?fieldSelector=metadata.name%3D192.168.51.81&re
Jul 14 10:02:58 controller kubelet-wrapper[1248]: E0714 10:02:58.380454 1248 reflector.go:188] pkg/kubelet/kubelet.go:378: Failed to list *api.Service: Get http://127.0.0.1:8080/api/v1/services?resourceVersion=0: dial tcp 127.0.0.1:80
Jul 14 10:02:58 controller kubelet-wrapper[1248]: E0714 10:02:58.381452 1248 reflector.go:188] pkg/kubelet/config/apiserver.go:44: Failed to list *api.Pod: Get http://127.0.0.1:8080/api/v1/pods?fieldSelector=spec.nodeName%3D192.168.51
Jul 14 10:02:58 controller kubelet-wrapper[1248]: E0714 10:02:58.628355 1248 reflector.go:188] pkg/kubelet/kubelet.go:386: Failed to list *api.Node: Get http://127.0.0.1:8080/api/v1/nodes?fieldSelector=metadata.name%3D192.168.51.81&re
lines 1-24/24 (END)

Logs::

Jul 14 10:30:10 controller kubelet-wrapper[1248]: E0714 10:30:10.516751 1248 reflector.go:188] pkg/kubelet/kubelet.go:378: Failed to list *api.Service: Get http://127.0.0.1:8080/api/v1/services?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
Jul 14 10:30:10 controller kubelet-wrapper[1248]: E0714 10:30:10.517665 1248 reflector.go:188] pkg/kubelet/config/apiserver.go:44: Failed to list *api.Pod: Get http://127.0.0.1:8080/api/v1/pods?fieldSelector=spec.nodeName%3D192.168.51.81&resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
Jul 14 10:30:10 controller kubelet-wrapper[1248]: E0714 10:30:10.663798 1248 reflector.go:188] pkg/kubelet/kubelet.go:386: Failed to list *api.Node: Get http://127.0.0.1:8080/api/v1/nodes?fieldSelector=metadata.name%3D192.168.51.81&resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
Jul 14 10:30:10 controller kubelet-wrapper[1248]: I0714 10:30:10.882697 1248 kubelet_node_status.go:204] Setting node annotation to enable volume controller attach/detach
Jul 14 10:30:10 controller kubelet-wrapper[1248]: I0714 10:30:10.883764 1248 kubelet_node_status.go:74] Attempting to register node 192.168.51.81
Jul 14 10:30:10 controller kubelet-wrapper[1248]: E0714 10:30:10.884014 1248 kubelet_node_status.go:98] Unable to register node "192.168.51.81" with API server: Post http://127.0.0.1:8080/api/v1/nodes: dial tcp 127.0.0.1:8080: getsockopt: connection refused
Jul 14 10:30:11 controller kubelet-wrapper[1248]: E0714 10:30:11.517400 1248 reflector.go:188] pkg/kubelet/kubelet.go:378: Failed to list *api.Service: Get http://127.0.0.1:8080/api/v1/services?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
Jul 14 10:30:11 controller kubelet-wrapper[1248]: E0714 10:30:11.518318 1248 reflector.go:188] pkg/kubelet/config/apiserver.go:44: Failed to list *api.Pod: Get http://127.0.0.1:8080/api/v1/pods?fieldSelector=spec.nodeName%3D192.168.51.81&resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
Jul 14 10:30:11 controller kubelet-wrapper[1248]: E0714 10:30:11.664582 1248 reflector.go:188] pkg/kubelet/kubelet.go:386: Failed to list *api.Node: Get http://127.0.0.1:8080/api/v1/nodes?fieldSelector=metadata.name%3D192.168.51.81&resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
Jul 14 10:30:11 controller kubelet-wrapper[1248]: E0714 10:30:11.857368 1248 cni.go:163] error updating cni config: No networks found in /etc/kubernetes/cni/net.d
Jul 14 10:30:12 controller kubelet-wrapper[1248]: E0714 10:30:12.518148 1248 reflector.go:188] pkg/kubelet/kubelet.go:378: Failed to list *api.Service: Get http://127.0.0.1:8080/api/v1/services?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
Jul 14 10:30:12 controller kubelet-wrapper[1248]: E0714 10:30:12.518996 1248 reflector.go:188] pkg/kubelet/config/apiserver.go:44: Failed to list *api.Pod: Get http://127.0.0.1:8080/api/v1/pods?fieldSelector=spec.nodeName%3D192.168.51.81&resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused

thanks