Failure when joining nodes to master

[colonelpopcorn]

I've been working with a Vagrant file to try and bootstrap a Kubernetes cluster using your role. I've set up a three machine file with one master and two nodes. At first, I had a lot of trouble with the script hanging at the join nodes to master step, but then I realized I was advertising the Kubernetes API on the NAT that Vagrant applies to all machines. After setting up a private network and changing the advertising address for Kubernetes I was able to get an actual error instead of hanging. Here's the link to my repo, and here's the output of the role:

TASK [kubernetes : include_tasks] **********************************************
skipping: [kube-master]
skipping: [kube-worker-1]
skipping: [kube-worker-2]

TASK [kubernetes : include_tasks] **********************************************
included: /home/jonathan/Documents/Projects/home-server/scripts/roles/kubernetes/tasks/setup-Debian.yml for kube-master, kube-worker-1, kube-worker-2

TASK [kubernetes : Ensure dependencies are installed.] *************************
ok: [kube-worker-1] => (item=[u'apt-transport-https', u'ca-certificates'])
ok: [kube-master] => (item=[u'apt-transport-https', u'ca-certificates'])
ok: [kube-worker-2] => (item=[u'apt-transport-https', u'ca-certificates'])

TASK [kubernetes : Add Kubernetes apt key.] ************************************
changed: [kube-worker-1]
changed: [kube-master]
changed: [kube-worker-2]

TASK [kubernetes : Add Kubernetes repository.] *********************************
changed: [kube-master]
changed: [kube-worker-2]
changed: [kube-worker-1]

TASK [kubernetes : Ensure dependencies are installed.] *************************
ok: [kube-worker-2]
ok: [kube-worker-1]
ok: [kube-master]

TASK [kubernetes : Install Kubernetes packages.] *******************************
changed: [kube-master] => (item={u'state': u'present', u'name': u'kubelet'})
changed: [kube-worker-1] => (item={u'state': u'present', u'name': u'kubelet'})
changed: [kube-master] => (item={u'state': u'present', u'name': u'kubeadm'})
ok: [kube-master] => (item={u'state': u'present', u'name': u'kubectl'})
changed: [kube-worker-2] => (item={u'state': u'present', u'name': u'kubelet'})
ok: [kube-master] => (item={u'state': u'present', u'name': u'kubernetes-cni'})
changed: [kube-worker-1] => (item={u'state': u'present', u'name': u'kubeadm'})
ok: [kube-worker-1] => (item={u'state': u'present', u'name': u'kubectl'})
ok: [kube-worker-1] => (item={u'state': u'present', u'name': u'kubernetes-cni'})
changed: [kube-worker-2] => (item={u'state': u'present', u'name': u'kubeadm'})
ok: [kube-worker-2] => (item={u'state': u'present', u'name': u'kubectl'})
ok: [kube-worker-2] => (item={u'state': u'present', u'name': u'kubernetes-cni'})

TASK [kubernetes : Configure KUBELET_EXTRA_ARGS.] ******************************
changed: [kube-worker-1]
changed: [kube-worker-2]
changed: [kube-master]

TASK [kubernetes : Reload systemd unit if args were changed.] ******************
changed: [kube-worker-2]
changed: [kube-worker-1]
changed: [kube-master]

TASK [kubernetes : Ensure kubelet is started and enabled at boot.] *************
ok: [kube-master]
ok: [kube-worker-1]
ok: [kube-worker-2]

TASK [kubernetes : Check if Kubernetes has already been initialized.] **********
ok: [kube-worker-1]
ok: [kube-worker-2]
ok: [kube-master]

TASK [kubernetes : include_tasks] **********************************************
skipping: [kube-worker-1]
skipping: [kube-worker-2]
included: /home/jonathan/Documents/Projects/home-server/scripts/roles/kubernetes/tasks/master-setup.yml for kube-master

TASK [kubernetes : Initialize Kubernetes master with kubeadm init.] ************
changed: [kube-master]

TASK [kubernetes : Print the init output to screen.] ***************************
skipping: [kube-master]

TASK [kubernetes : Ensure .kube directory exists.] *****************************
changed: [kube-master]

TASK [kubernetes : Symlink the kubectl admin.conf to ~/.kube/conf.] ************
changed: [kube-master]

TASK [kubernetes : Configure Flannel networking.] ******************************
changed: [kube-master] => (item=kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/k8s-manifests/kube-flannel-rbac.yml)
changed: [kube-master] => (item=kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml)

TASK [kubernetes : Allow pods on master node (if configured).] *****************
changed: [kube-master]

TASK [kubernetes : Check if Kubernetes Dashboard UI service already exists.] ***
ok: [kube-master]

TASK [kubernetes : Enable the Kubernetes Web Dashboard UI (if configured).] ****
skipping: [kube-master]

TASK [kubernetes : Get the kubeadm join command from the Kubernetes master.] ***
ok: [kube-master]

TASK [kubernetes : include_tasks] **********************************************
skipping: [kube-master]
included: /home/jonathan/Documents/Projects/home-server/scripts/roles/kubernetes/tasks/node-setup.yml for kube-worker-1, kube-worker-2

TASK [kubernetes : Join node to Kubernetes master] *****************************
fatal: [kube-worker-1]: FAILED! => {"changed": true, "cmd": "kubeadm join 10.0.0.10:6443 --token 3w17ow.5vh1zdgcg6lgzgtj --discovery-token-ca-cert-hash sha256:f8276059fb2b765a55b4723cfb5e7ba413c021c58c5eb81bcdccac2933a730c0", "delta": "0:00:10.863798", "end": "2018-07-09 11:22:15.461549", "msg": "non-zero return code", "rc": 1, "start": "2018-07-09 11:22:04.597751", "stderr": "\t[WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh] or no builtin kernel ipvs support: map[ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{}]\nyou can solve this problem with following methods:\n 1. Run 'modprobe -- ' to load missing kernel modules;\n2. Provide the missing builtin kernel ipvs support\n\nI0709 11:22:04.695777    7794 kernel_validator.go:81] Validating kernel version\nI0709 11:22:04.695970    7794 kernel_validator.go:96] Validating kernel config\n\t[WARNING SystemVerification]: docker version is greater than the most recently validated version.Docker version: 18.03.1-ce. Max validated version: 17.03\nconfigmaps \"kubelet-config-1.11\" is forbidden: User \"system:bootstrap:3w17ow\" cannot get configmaps in the namespace \"kube-system\"", "stderr_lines": ["\t[WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh] or no builtin kernel ipvs support: map[ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{}]", "you can solve this problem with following methods:", " 1. Run 'modprobe -- ' to load missing kernel modules;", "2. Provide the missing builtin kernel ipvs support", "", "I0709 11:22:04.695777    7794 kernel_validator.go:81] Validating kernel version", "I0709 11:22:04.695970    7794 kernel_validator.go:96] Validating kernel config", "\t[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.03.1-ce. Max validated version: 17.03", "configmaps \"kubelet-config-1.11\" is forbidden: User \"system:bootstrap:3w17ow\" cannot get configmaps in the namespace \"kube-system\""], "stdout": "[preflight] running pre-flight checks\n[discovery] Trying to connect to API Server \"10.0.0.10:6443\"\n[discovery] Created cluster-info discovery client, requesting info from \"https://10.0.0.10:6443\"\n[discovery] Failed to connect to API Server \"10.0.0.10:6443\": token id \"3w17ow\" is invalid for this cluster or it has expired. Use \"kubeadm token create\" on the master node to creating a new valid token\n[discovery] Trying to connect to API Server \"10.0.0.10:6443\"\n[discovery] Created cluster-info discovery client, requesting info from \"https://10.0.0.10:6443\"\n[discovery] Failed to connect to API Server \"10.0.0.10:6443\": token id \"3w17ow\" is invalid for this cluster or it has expired. Use \"kubeadm token create\" on the master node to creating a new valid token\n[discovery] Trying to connect to API Server \"10.0.0.10:6443\"\n[discovery] Created cluster-info discovery client, requesting info from \"https://10.0.0.10:6443\"\n[discovery] Requesting info from \"https://10.0.0.10:6443\" again to validate TLS against the pinned public key\n[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server \"10.0.0.10:6443\"\n[discovery] Successfully established connection with API Server \"10.0.0.10:6443\"\n[kubelet] Downloading configuration for the kubelet from the \"kubelet-config-1.11\" ConfigMap inthe kube-system namespace", "stdout_lines": ["[preflight] running pre-flight checks", "[discovery] Trying to connect to API Server \"10.0.0.10:6443\"", "[discovery] Created cluster-info discovery client, requesting info from \"https://10.0.0.10:6443\"", "[discovery] Failed to connect to API Server \"10.0.0.10:6443\": token id \"3w17ow\" is invalid for this cluster or it has expired. Use \"kubeadm token create\" on the master node to creating a new valid token", "[discovery] Trying to connect to API Server \"10.0.0.10:6443\"", "[discovery] Created cluster-info discovery client, requesting info from \"https://10.0.0.10:6443\"", "[discovery] Failed to connect to API Server \"10.0.0.10:6443\": token id \"3w17ow\" is invalid for this cluster or it has expired. Use \"kubeadm token create\" on the master node to creating a new valid token", "[discovery] Trying to connect to API Server \"10.0.0.10:6443\"", "[discovery] Created cluster-info discovery client, requesting info from \"https://10.0.0.10:6443\"", "[discovery] Requesting info from \"https://10.0.0.10:6443\" again to validate TLS against the pinned public key", "[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server \"10.0.0.10:6443\"", "[discovery] Successfully established connection with API Server \"10.0.0.10:6443\"", "[kubelet] Downloading configuration for the kubelet from the \"kubelet-config-1.11\" ConfigMap in the kube-system namespace"]}
fatal: [kube-worker-2]: FAILED! => {"changed": true, "cmd": "kubeadm join 10.0.0.10:6443 --token 3w17ow.5vh1zdgcg6lgzgtj --discovery-token-ca-cert-hash sha256:f8276059fb2b765a55b4723cfb5e7ba413c021c58c5eb81bcdccac2933a730c0", "delta": "0:00:10.880070", "end": "2018-07-09 11:22:14.974144", "msg": "non-zero return code", "rc": 1, "start": "2018-07-09 11:22:04.094074", "stderr": "\t[WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs_sh ip_vs ip_vs_rr ip_vs_wrr] or no builtin kernel ipvs support: map[ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{} ip_vs:{}]\nyou can solve this problem with following methods:\n 1. Run 'modprobe -- ' to load missing kernel modules;\n2. Provide the missing builtin kernel ipvs support\n\nI0709 11:22:04.206006    7786 kernel_validator.go:81] Validating kernel version\nI0709 11:22:04.206166    7786 kernel_validator.go:96] Validating kernel config\n\t[WARNING SystemVerification]: docker version is greater than the most recently validated version.Docker version: 18.03.1-ce. Max validated version: 17.03\nconfigmaps \"kubelet-config-1.11\" is forbidden: User \"system:bootstrap:3w17ow\" cannot get configmaps in the namespace \"kube-system\"", "stderr_lines": ["\t[WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs_sh ip_vs ip_vs_rr ip_vs_wrr] or no builtin kernel ipvs support: map[ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{} ip_vs:{}]", "you can solve this problem with following methods:", " 1. Run 'modprobe -- ' to load missing kernel modules;", "2. Provide the missing builtin kernel ipvs support", "", "I0709 11:22:04.206006    7786 kernel_validator.go:81] Validating kernel version", "I0709 11:22:04.206166    7786 kernel_validator.go:96] Validating kernel config", "\t[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 18.03.1-ce. Max validated version: 17.03", "configmaps \"kubelet-config-1.11\" is forbidden: User \"system:bootstrap:3w17ow\" cannot get configmaps in the namespace \"kube-system\""], "stdout": "[preflight] running pre-flight checks\n[discovery] Trying to connect to API Server \"10.0.0.10:6443\"\n[discovery] Created cluster-info discovery client, requesting info from \"https://10.0.0.10:6443\"\n[discovery] Failed to connect to API Server \"10.0.0.10:6443\": token id \"3w17ow\" is invalid for this cluster or it has expired. Use \"kubeadm token create\" on the master node to creating a new valid token\n[discovery] Trying to connect to API Server \"10.0.0.10:6443\"\n[discovery] Created cluster-info discovery client, requesting info from \"https://10.0.0.10:6443\"\n[discovery] Failed to connect to API Server \"10.0.0.10:6443\": token id \"3w17ow\" is invalid for this cluster or it has expired. Use \"kubeadm token create\" on the master node to creating a new valid token\n[discovery] Trying to connect to API Server \"10.0.0.10:6443\"\n[discovery] Created cluster-info discovery client, requesting info from \"https://10.0.0.10:6443\"\n[discovery] Requesting info from \"https://10.0.0.10:6443\" again to validate TLS against the pinned public key\n[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server \"10.0.0.10:6443\"\n[discovery] Successfully established connection with API Server \"10.0.0.10:6443\"\n[kubelet] Downloading configuration for the kubelet from the \"kubelet-config-1.11\" ConfigMap inthe kube-system namespace", "stdout_lines": ["[preflight] running pre-flight checks", "[discovery] Trying to connect to API Server \"10.0.0.10:6443\"", "[discovery] Created cluster-info discovery client, requesting info from \"https://10.0.0.10:6443\"", "[discovery] Failed to connect to API Server \"10.0.0.10:6443\": token id \"3w17ow\" is invalid for this cluster or it has expired. Use \"kubeadm token create\" on the master node to creating a new valid token", "[discovery] Trying to connect to API Server \"10.0.0.10:6443\"", "[discovery] Created cluster-info discovery client, requesting info from \"https://10.0.0.10:6443\"", "[discovery] Failed to connect to API Server \"10.0.0.10:6443\": token id \"3w17ow\" is invalid for this cluster or it has expired. Use \"kubeadm token create\" on the master node to creating a new valid token", "[discovery] Trying to connect to API Server \"10.0.0.10:6443\"", "[discovery] Created cluster-info discovery client, requesting info from \"https://10.0.0.10:6443\"", "[discovery] Requesting info from \"https://10.0.0.10:6443\" again to validate TLS against the pinned public key", "[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server \"10.0.0.10:6443\"", "[discovery] Successfully established connection with API Server \"10.0.0.10:6443\"", "[kubelet] Downloading configuration for the kubelet from the \"kubelet-config-1.11\" ConfigMap in the kube-system namespace"]}

RUNNING HANDLER [kubernetes : restart kubelet] *********************************
changed: [kube-master]
        to retry, use: --limit @/home/jonathan/Documents/Projects/home-server/scripts/bootstrap.retry

PLAY RECAP *********************************************************************
kube-master                : ok=31   changed=16   unreachable=0    failed=0
kube-worker-1              : ok=23   changed=10   unreachable=0    failed=1
kube-worker-2              : ok=23   changed=10   unreachable=0    failed=1

Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

I think I might need to change OS or add a package, but I'm kind of lost at this point.
EDIT: Just read the rest of the error output and noticed that the token isn't working. Is this a formatting issue? Should I print the join command fully and see what's happening with it?

[tonyppe]

I am also stuck on this point with what looks like the same issue. I'm trying to use Centos 7 now. Ubuntu 16 failed also but at an earlier part.

[davidcodesido]

I think this is maybe related: kubernetes/kubeadm#907

I fixed this issue by making sure that the versions are the same everywhere. In particular I had to change the variable "kubernetes_version" to 'stable-1.11'

[colonelpopcorn]

Thanks @davidcodesido, stable-1.11 worked. Should the version be updated in the defaults, @geerlingguy?

[geerlingguy]

@colonelpopcorn - Would definitely accept a PR! I already have a different version locked in in the infra where it is necessary, so the role following the mainline stable release is a good idea.

[vicsun2007]

I got the same problem and resolved it after install the same version(kubelet-1.10.1) with master

[colonelpopcorn]

PR #12 fixed this issue. Latest version should be able to stand up a multi-node kubernetes cluster with very little pain and suffering! Thanks all for pointing out solutions!

[geerlingguy]

@colonelpopcorn - Thanks again!

[geerlingguy]

Just released a 1.0.0 since this role is at least somewhat stable in my testing. Any major architecture changes will get a version bump.