error: metrics not available yet - No error in the logs

[jibinrajck]

What happened: kubectl top pods returning
error: Metrics not available for pod default/awscli, age: 177h19m9.789235105s

& k top nodes returns

error: metrics not available yet

What you expected to happen: Should retrun valid responses

Anything else we need to know?:

Logs :

[ec2-user@10.161.133.102 2024-02-05_09:59:16 UTC ~]$ k top node --v=8
I0205 10:00:43.902192   21565 loader.go:373] Config loaded from file:  /home/ec2-user/.kube/config
I0205 10:00:43.903956   21565 round_trippers.go:463] GET https://xxxxxxxxxxxxxxxxxxxxx.gr7.ap-southeast-2.eks.amazonaws.com/api
I0205 10:00:43.903996   21565 round_trippers.go:469] Request Headers:
I0205 10:00:43.904014   21565 round_trippers.go:473]     Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json
I0205 10:00:43.904030   21565 round_trippers.go:473]     User-Agent: kubectl/v1.26.7 (linux/amd64) kubernetes/89a3d86
I0205 10:00:44.054485   21565 round_trippers.go:574] Response Status: 200 OK in 150 milliseconds
I0205 10:00:44.057617   21565 round_trippers.go:577] Response Headers:
I0205 10:00:44.057727   21565 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: 70464fe9-5fd3-4a02-abad-d4bdf1640edb
I0205 10:00:44.057747   21565 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 934dbb0f-5e8b-4602-baaa-b960847fbfb2
I0205 10:00:44.057763   21565 round_trippers.go:580]     Content-Length: 170
I0205 10:00:44.057780   21565 round_trippers.go:580]     Date: Mon, 05 Feb 2024 10:00:44 GMT
I0205 10:00:44.057794   21565 round_trippers.go:580]     Audit-Id: de54b295-eb5c-4091-933f-65ad60069091
I0205 10:00:44.057806   21565 round_trippers.go:580]     Cache-Control: no-cache, private
I0205 10:00:44.057821   21565 round_trippers.go:580]     Content-Type: application/json
I0205 10:00:44.057857   21565 request.go:1171] Response Body: {"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"ip-123-45-678-98.ap-southeast-2.compute.internal:443"}]}
I0205 10:00:44.058534   21565 round_trippers.go:463] GET https://xxxxxxxxxxxxxxxxxxxxx.gr7.ap-southeast-2.eks.amazonaws.com/apis
I0205 10:00:44.058629   21565 round_trippers.go:469] Request Headers:
I0205 10:00:44.058650   21565 round_trippers.go:473]     Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json
I0205 10:00:44.058667   21565 round_trippers.go:473]     User-Agent: kubectl/v1.26.7 (linux/amd64) kubernetes/89a3d86
I0205 10:00:44.059521   21565 round_trippers.go:574] Response Status: 200 OK in 0 milliseconds
I0205 10:00:44.059546   21565 round_trippers.go:577] Response Headers:
I0205 10:00:44.059563   21565 round_trippers.go:580]     Audit-Id: 8c14f103-4ffd-42e1-b60f-8d34d871cbcd
I0205 10:00:44.059588   21565 round_trippers.go:580]     Cache-Control: no-cache, private
I0205 10:00:44.059604   21565 round_trippers.go:580]     Content-Type: application/json
I0205 10:00:44.059619   21565 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: 70464fe9-5fd3-4a02-abad-d4bdf1640edb
I0205 10:00:44.059636   21565 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 934dbb0f-5e8b-4602-baaa-b960847fbfb2
I0205 10:00:44.059653   21565 round_trippers.go:580]     Date: Mon, 05 Feb 2024 10:00:44 GMT
I0205 10:00:44.060253   21565 request.go:1171] Response Body: {"kind":"APIGroupList","apiVersion":"v1","groups":[{"name":"apiregistration.k8s.io","versions":[{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"apiregistration.k8s.io/v1","version":"v1"}},{"name":"apps","versions":[{"groupVersion":"apps/v1","version":"v1"}],"preferredVersion":{"groupVersion":"apps/v1","version":"v1"}},{"name":"events.k8s.io","versions":[{"groupVersion":"events.k8s.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"events.k8s.io/v1","version":"v1"}},{"name":"authentication.k8s.io","versions":[{"groupVersion":"authentication.k8s.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"authentication.k8s.io/v1","version":"v1"}},{"name":"authorization.k8s.io","versions":[{"groupVersion":"authorization.k8s.io/v1","version":"v1"}],"preferredVersion":{"groupVersion":"authorization.k8s.io/v1","version":"v1"}},{"name":"autoscaling","versions":[{"groupVersion":"autoscaling/v2","version":"v2"},{"groupVersion":"autoscaling/v1","version":"v1"}], [truncated 4396 chars]
I0205 10:00:44.062517   21565 round_trippers.go:463] GET https://xxxxxxxxxxxxxxxxxxxxx.gr7.ap-southeast-2.eks.amazonaws.com/apis/metrics.k8s.io/v1beta1/nodes
I0205 10:00:44.062611   21565 round_trippers.go:469] Request Headers:
I0205 10:00:44.062632   21565 round_trippers.go:473]     Accept: application/vnd.kubernetes.protobuf, */*
I0205 10:00:44.062651   21565 round_trippers.go:473]     User-Agent: kubectl/v1.26.7 (linux/amd64) kubernetes/89a3d86
I0205 10:00:44.072123   21565 round_trippers.go:574] Response Status: 200 OK in 9 milliseconds
I0205 10:00:44.073105   21565 round_trippers.go:577] Response Headers:
I0205 10:00:44.073131   21565 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 934dbb0f-5e8b-4602-baaa-b960847fbfb2
I0205 10:00:44.073147   21565 round_trippers.go:580]     Content-Length: 61
I0205 10:00:44.073162   21565 round_trippers.go:580]     Audit-Id: ee0f3e0e-824f-4368-83ff-d888f818f2ef
I0205 10:00:44.073175   21565 round_trippers.go:580]     Cache-Control: no-cache, private
I0205 10:00:44.073188   21565 round_trippers.go:580]     Cache-Control: no-cache, private
I0205 10:00:44.073202   21565 round_trippers.go:580]     Content-Type: application/vnd.kubernetes.protobuf
I0205 10:00:44.073219   21565 round_trippers.go:580]     Date: Mon, 05 Feb 2024 10:00:44 GMT
I0205 10:00:44.073234   21565 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: 70464fe9-5fd3-4a02-abad-d4bdf1640edb
I0205 10:00:44.073272   21565 request.go:1169] Response Body:
00000000  6b 38 73 00 0a 29 0a 16  6d 65 74 72 69 63 73 2e  |k8s..)..metrics.|
00000010  6b 38 73 2e 69 6f 2f 76  31 62 65 74 61 31 12 0f  |k8s.io/v1beta1..|
00000020  4e 6f 64 65 4d 65 74 72  69 63 73 4c 69 73 74 12  |NodeMetricsList.|
00000030  08 0a 06 0a 00 12 00 1a  00 1a 00 22 00           |...........".|
error: metrics not available yet

[ec2-user@10.161.133.102 2024-02-05_10:11:27 UTC ~]$ kn logs metrics-server-5c5f997c95-gkt9t
I0205 09:57:11.782601       1 serving.go:374] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
I0205 09:57:12.211023       1 handler.go:275] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
I0205 09:57:12.320721       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0205 09:57:12.320934       1 shared_informer.go:311] Waiting for caches to sync for RequestHeaderAuthRequestController
I0205 09:57:12.320967       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0205 09:57:12.321004       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0205 09:57:12.321104       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0205 09:57:12.321116       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0205 09:57:12.322559       1 secure_serving.go:213] Serving securely on [::]:10250
I0205 09:57:12.322618       1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key"
I0205 09:57:12.322784       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0205 09:57:12.421066       1 shared_informer.go:318] Caches are synced for RequestHeaderAuthRequestController
I0205 09:57:12.421150       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0205 09:57:12.421198       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-fil

Environment:

• Kubernetes distribution (GKE, EKS, Kubeadm, the hard way, etc.): EKS

• Container Network Setup (flannel, calico, etc.): AWS VPC CNI

• Kubernetes version (use kubectl version): 1.26

• Metrics Server manifest
  kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

spoiler for Metrics Server manifest:

• Kubelet config:

spoiler for Kubelet config:

• Metrics server logs:

spoiler for Metrics Server logs:

• Status of Metrics API:

spolier for Status of Metrics API:

kubectl describe apiservice v1beta1.metrics.k8s.io

/kind bug

[dgrisonnet]

/kind support
/triage accepted
/assign @yangjunmyfm192085

[yangjunmyfm192085]

hi, @jibinrajck It seems that metrics server does not obtain the metrics information of pods or nodes. Can you look at the logs of metrics server?