[discovery] Failed to request cluster info, will try again: [Unauthorized]

[shenshouer]

kubeadm join --token f2f906.2d84a4696fe861d7 10.23.27.99:443 --discovery-token-ca-cert-hash sha256:ad3d1777b71be96e9fa19552a6b2e6c621b0b8aaa1bf970f40d0ca9a01320e3b
[preflight] Running pre-flight checks.
	[WARNING SystemVerification]: docker version is greater than the most recently validated version. Docker version: 17.06.2-ce. Max validated version: 17.03
	[WARNING Hostname]: hostname "l23-80-2" could not be reached
	[WARNING Hostname]: hostname "l23-80-2" lookup l23-80-2 on 192.168.252.25:53: no such host
	[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
	[WARNING FileExisting-crictl]: crictl not found in system path
[discovery] Trying to connect to API Server "10.23.27.99:443"
[discovery] Created cluster-info discovery client, requesting info from "https://10.23.27.99:443"
[discovery] Failed to request cluster info, will try again: [Unauthorized]
[discovery] Failed to request cluster info, will try again: [Unauthorized]
[discovery] Failed to request cluster info, will try again: [Unauthorized]
[discovery] Failed to request cluster info, will try again: [Unauthorized]
[discovery] Failed to request cluster info, will try again: [Unauthorized]
[discovery] Failed to request cluster info, will try again: [Unauthorized]

the version of all component is v1.9.1

Anyone who can help me ?

[embik]

Would you mind sharing your solution @shenshouer ?

[foxyriver]

@embik config --anonymous-auth=false in kube-apiserver will result this issue

[yaegel]

I've got the same issue with Kubeadm 1.11.3-00 but my --anonymous-auth is true. This is the output of kubectl describe pod kube-apiserver- -n kube-system

Name:               kube-apiserver-<master-host>
Namespace:          kube-system
Priority:           2000000000
PriorityClassName:  system-cluster-critical
Node:               <master-host>/*.*.*.*
Start Time:         Wed, 19 Sep 2018 11:36:43 -0600
Labels:             component=kube-apiserver
                    tier=control-plane
Annotations:        kubernetes.io/config.hash=d76abd05f9b4afc6752e1cb8fff8db29
                    kubernetes.io/config.mirror=d76abd05f9b4afc6752e1cb8fff8db29
                    kubernetes.io/config.seen=2019-02-28T13:49:43.049237451-07:00
                    kubernetes.io/config.source=file
                    scheduler.alpha.kubernetes.io/critical-pod=
Status:             Running
IP:                 *.*.*.*
Containers:
  kube-apiserver:
    Container ID:  docker://419208be3100d575b1946e56546d2424bbee5c7e5ac76860071e7aa28b49ae5d
    Image:         k8s.gcr.io/kube-apiserver-amd64:v1.11.3
    Image ID:      docker-pullable://k8s.gcr.io/kube-apiserver-amd64@sha256:956bea8c139620c9fc823fb81ff9b5647582b53bd33904302987d56ab24fc187
    Port:          <none>
    Host Port:     <none>
    Command:
      kube-apiserver
      --authorization-mode=AlwaysAllow
      --advertise-address=*.*.*.*
      --allow-privileged=true
      --anonymous-auth=true
      --client-ca-file=/etc/kubernetes/pki/ca.crt
      --disable-admission-plugins=PersistentVolumeLabel
      --enable-admission-plugins=NodeRestriction
      --enable-bootstrap-token-auth=true
      --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
      --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
      --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
      --etcd-servers=https://127.0.0.1:2379
      --insecure-port=0
      --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
      --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
      --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
      --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
      --requestheader-allowed-names=front-proxy-client
      --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
      --requestheader-extra-headers-prefix=X-Remote-Extra-
      --requestheader-group-headers=X-Remote-Group
      --requestheader-username-headers=X-Remote-User
      --secure-port=6443
      --service-account-key-file=/etc/kubernetes/pki/sa.pub
      --service-cluster-ip-range=10.96.0.0/12
      --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
      --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    State:          Running
      Started:      Thu, 28 Feb 2019 13:49:44 -0700
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        250m
    Liveness:     http-get https://*.*.*.*:6443/healthz delay=15s timeout=15s period=10s #success=1 #failure=8
    Environment:  <none>
    Mounts:
      /etc/ca-certificates from etc-ca-certificates (ro)
      /etc/kubernetes/pki from k8s-certs (ro)
      /etc/pki from etc-pki (ro)
      /etc/ssl/certs from ca-certs (ro)
      /usr/local/share/ca-certificates from usr-local-share-ca-certificates (ro)
      /usr/share/ca-certificates from usr-share-ca-certificates (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  k8s-certs:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/pki
    HostPathType:  DirectoryOrCreate
  ca-certs:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/ssl/certs
    HostPathType:  DirectoryOrCreate
  etc-pki:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/pki
    HostPathType:  DirectoryOrCreate
  usr-share-ca-certificates:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/share/ca-certificates
    HostPathType:  DirectoryOrCreate
  usr-local-share-ca-certificates:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/local/share/ca-certificates
    HostPathType:  DirectoryOrCreate
  etc-ca-certificates:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/ca-certificates
    HostPathType:  DirectoryOrCreate
QoS Class:         Burstable
Node-Selectors:    <none>
Tolerations:       :NoExecute
Events:            <none>

And here is the output when I try to join after creating a new token with the --print-join-command and -v9 flags:

I0304 09:13:14.427364 3176882 join.go:226] [join] found NodeName empty
I0304 09:13:14.427443 3176882 join.go:227] [join] considered OS hostname as NodeName
[preflight] running pre-flight checks
I0304 09:13:14.427578 3176882 join.go:238] [preflight] running various checks on all nodes
I0304 09:13:14.427630 3176882 checks.go:253] validating the existence and emptiness of directory /etc/kubernetes/manifests
I0304 09:13:14.427699 3176882 checks.go:291] validating the existence of file /etc/kubernetes/pki/ca.crt
I0304 09:13:14.427717 3176882 checks.go:291] validating the existence of file /etc/kubernetes/kubelet.conf
I0304 09:13:14.427732 3176882 checks.go:291] validating the existence of file /etc/kubernetes/bootstrap-kubelet.conf
I0304 09:13:14.427753 3176882 kernelcheck_linux.go:45] validating the kernel module IPVS required exists in machine or not
	[WARNING RequiredIPVSKernelModulesAvailable]: the IPVS proxier will not be used, because the following required kernel modules are not loaded: [ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh] or no builtin kernel ipvs support: map[ip_vs:{} ip_vs_rr:{} ip_vs_wrr:{} ip_vs_sh:{} nf_conntrack_ipv4:{}]
you can solve this problem with following methods:
 1. Run 'modprobe -- ' to load missing kernel modules;
2. Provide the missing builtin kernel ipvs support

I0304 09:13:14.431606 3176882 checks.go:138] validating if the service is enabled and active
I0304 09:13:14.448891 3176882 checks.go:340] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
I0304 09:13:14.448970 3176882 checks.go:340] validating the contents of file /proc/sys/net/ipv4/ip_forward
I0304 09:13:14.449017 3176882 checks.go:653] validating whether swap is enabled or not
I0304 09:13:14.449083 3176882 checks.go:381] validating the presence of executable crictl
I0304 09:13:14.449131 3176882 checks.go:381] validating the presence of executable ip
I0304 09:13:14.449165 3176882 checks.go:381] validating the presence of executable iptables
I0304 09:13:14.449195 3176882 checks.go:381] validating the presence of executable mount
I0304 09:13:14.449224 3176882 checks.go:381] validating the presence of executable nsenter
I0304 09:13:14.449252 3176882 checks.go:381] validating the presence of executable ebtables
I0304 09:13:14.449281 3176882 checks.go:381] validating the presence of executable ethtool
I0304 09:13:14.449307 3176882 checks.go:381] validating the presence of executable socat
I0304 09:13:14.449346 3176882 checks.go:381] validating the presence of executable tc
I0304 09:13:14.449377 3176882 checks.go:381] validating the presence of executable touch
I0304 09:13:14.449402 3176882 checks.go:523] running all checks
I0304 09:13:14.451248 3176882 kernel_validator.go:81] Validating kernel version
I0304 09:13:14.451372 3176882 kernel_validator.go:96] Validating kernel config
I0304 09:13:14.780475 3176882 checks.go:411] checking whether the given node name is reachable using net.LookupHost
I0304 09:13:14.780711 3176882 checks.go:622] validating kubelet version
I0304 09:13:14.951219 3176882 checks.go:138] validating if the service is enabled and active
I0304 09:13:14.966615 3176882 checks.go:216] validating availability of port 10250
I0304 09:13:14.966764 3176882 checks.go:438] validating if the connectivity type is via proxy or direct
I0304 09:13:14.966809 3176882 join.go:251] [join] retrieving KubeConfig objects
[discovery] Trying to connect to API Server "*.*.*.*:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://*.*.*.*:6443"
I0304 09:13:14.967746 3176882 round_trippers.go:386] curl -k -v -XGET  -H "User-Agent: kubeadm/v1.11.3 (linux/amd64) kubernetes/a452946" -H "Accept: application/json, */*" 'https://*.*.*.*:6443/api/v1/namespaces/kube-public/configmaps/cluster-info'
I0304 09:13:14.974195 3176882 round_trippers.go:405] GET https://*.*.*.*:6443/api/v1/namespaces/kube-public/configmaps/cluster-info 401 Unauthorized in 6 milliseconds
I0304 09:13:14.974229 3176882 round_trippers.go:411] Response Headers:
I0304 09:13:14.974238 3176882 round_trippers.go:414]     Content-Type: application/json
I0304 09:13:14.974250 3176882 round_trippers.go:414]     Content-Length: 129
I0304 09:13:14.974260 3176882 round_trippers.go:414]     Date: Mon, 04 Mar 2019 16:13:14 GMT
I0304 09:13:14.974302 3176882 request.go:897] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
[discovery] Failed to request cluster info, will try again: [Unauthorized]
I0304 09:13:19.975419 3176882 round_trippers.go:386] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubeadm/v1.11.3 (linux/amd64) kubernetes/a452946" 'https://*.*.*.*:6443/api/v1/namespaces/kube-public/configmaps/cluster-info'
I0304 09:13:19.976262 3176882 round_trippers.go:405] GET https://*.*.*.*:6443/api/v1/namespaces/kube-public/configmaps/cluster-info 401 Unauthorized in 0 milliseconds
I0304 09:13:19.976288 3176882 round_trippers.go:411] Response Headers:
I0304 09:13:19.976300 3176882 round_trippers.go:414]     Content-Type: application/json
I0304 09:13:19.976313 3176882 round_trippers.go:414]     Content-Length: 129
I0304 09:13:19.976325 3176882 round_trippers.go:414]     Date: Mon, 04 Mar 2019 16:13:19 GMT
I0304 09:13:19.976366 3176882 request.go:897] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
[discovery] Failed to request cluster info, will try again: [Unauthorized]

Am I missing something, or should this join up without error?