Failed to scrape node" err="Get \"https://10.100.93.58:10250/metrics/resource\": context deadline exceeded"

[MahiraTechnology]

What happened:
Looks like pods are not scaling based on the load which is causing the pods to restart

What you expected to happen:
HPA should scale based on the load
Anything else we need to know?:

Environment:
Running on EKS 1.27
metrics-server 0.6.3

• Kubernetes distribution (GKE, EKS, Kubeadm, the hard way, etc.):

• Container Network Setup (flannel, calico, etc.):

• Kubernetes version (use kubectl version):

• Metrics Server manifest

spoiler for Metrics Server manifest:

Using helm chart

• Kubelet config:

spoiler for Kubelet config:

• Metrics server logs:

spoiler for Metrics Server logs:

I1025 19:47:59.616036 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
E1025 19:48:28.004348 1 scraper.go:140] "Failed to scrape node" err="Get "https://10.100.55.152:10250/metrics/resource\": context deadline exceeded" node="ip-10-100-55-152.ca-central-1.compute.internal"
E1025 19:48:58.004680 1 scraper.go:140] "Failed to scrape node" err="Get "https://10.100.48.155:10250/metrics/resource\": dial tcp 10.100.48.155:10250: i/o timeout" node="ip-10-100-48-155.ca-central-1.compute.internal"
E1025 19:49:13.005190 1 scraper.go:140] "Failed to scrape node" err="Get "https://10.100.48.155:10250/metrics/resource\": context deadline exceeded" node="ip-10-100-48-155.ca-central-1.compute.internal"
E1025 19:49:28.003975 1 scraper.go:140] "Failed to scrape node" err="Get "https://10.100.48.155:10250/metrics/resource\": context deadline exceeded" node="ip-10-100-48-155.ca-central-1.compute.internal"
E1025 19:53:29.599618 1 scraper.go:140] "Failed to scrape node" err="Get "https://10.100.78.163:10250/metrics/resource\": remote error: tls: internal error" node="ip-10-100-78-163.ca-central-1.compute.internal"
E1025 19:54:44.588439 1 scraper.go:140] "Failed to scrape node" err="Get "https://10.100.50.210:10250/metrics/resource\": remote error: tls: internal error" node="ip-10-100-50-210.ca-central-1.compute.internal"
E1025 19:55:28.004773 1 scraper.go:140] "Failed to scrape node" err="Get "https://10.100.73.41:10250/metrics/resource\": context deadline exceeded" node="ip-10-100-73-41.ca-central-1.compute.internal"

• Status of Metrics API:

spolier for Status of Metrics API:

kubectl describe apiservice v1beta1.metrics.k8s.io

kubectl describe apiservices v1beta1.metrics.k8s.io
Name: v1beta1.metrics.k8s.io
Namespace:
Labels: app.kubernetes.io/instance=metrics-server
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=metrics-server
app.kubernetes.io/version=0.6.3
helm.sh/chart=metrics-server-3.10.0
Annotations: meta.helm.sh/release-name: metrics-server
meta.helm.sh/release-namespace: kube-system
API Version: apiregistration.k8s.io/v1
Kind: APIService
Metadata:
Creation Timestamp: 2023-07-03T08:59:27Z
Resource Version: 82108513
UID: de273b86-9ba6-4d8d-929c-b972d87717e1
Spec:
Group: metrics.k8s.io
Group Priority Minimum: 100
Insecure Skip TLS Verify: true
Service:
Name: metrics-server
Namespace: kube-system
Port: 443
Version: v1beta1
Version Priority: 100
Status:
Conditions:
Last Transition Time: 2023-10-25T19:48:23Z
Message: all checks passed
Reason: Passed
Status: True
Type: Available
Events:

/kind bug

[brosef]

Are you using any CNI plugin (calico, weave, vpc-cni, etc.)? If so, setting hostnetwork: true in your deployment might help.

[MahiraTechnology]

Yes using aws VPC-cni. Let me try the setting up hostnetwork: true in deployment of metrics server and update the observation here.

[brosef]

@MahiraTechnology Let me know how it goes - we're on EKS and noticed in newer versions of the vpc-cni plugin, there was a communication breakdown somewhere between the pod VPC (where metrics-server runs), the node VPC, and the control plane. After setting hostnetwork: true, everything worked A-Ok.

[MahiraTechnology]

@brosef i tried to deploy the Metrics server with hostnetwork: true, starting seeing below issue.

panic: failed to create listener: failed to listen on 0.0.0.0:10250: listen tcp 0.0.0.0:10250: bind: address already in use
goroutine 1 [running]:
main.main()
/go/src/sigs.k8s.io/metrics-server/cmd/metrics-server/metrics-server.go:37 +0xa5

[brosef]

you probably have to change the port to something else, 10250 will clash with the kubelet API port. try setting containerPort: 4443

[MahiraTechnology]

@brosef i deployed with port 4443 still i am seeing the same issue in metrics server pod.

I1026 18:09:39.580732 1 serving.go:342] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
I1026 18:09:40.087766 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I1026 18:09:40.087788 1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I1026 18:09:40.087790 1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I1026 18:09:40.087800 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1026 18:09:40.087777 1 configmap_cafile_content.go:201] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I1026 18:09:40.087969 1 secure_serving.go:267] Serving securely on [::]:4443
I1026 18:09:40.088007 1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I1026 18:09:40.087979 1 dynamic_serving_content.go:131] "Starting controller" name="serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key"
I1026 18:09:40.087993 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
W1026 18:09:40.088063 1 shared_informer.go:372] The sharedIndexInformer has started, run more than once is not allowed
I1026 18:09:40.188199 1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController
I1026 18:09:40.188217 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1026 18:09:40.188231 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
E1026 18:23:38.585351 1 scraper.go:140] "Failed to scrape node" err="Get "https://10.100.69.44:10250/metrics/resource\": context deadline exceeded" node="ip-10-100-69-44.ca-central-1.compute.internal"

[brosef]

check your security group and firewall rules. ensure tcp 10250 is open between nodes

[MahiraTechnology]

@brosef
i see below error in events

network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

[brosef]

that could mean one of many things. try running through this article: https://repost.aws/knowledge-center/eks-cni-plugin-troubleshooting

[MahiraTechnology]

@brosef i went through above shared link , everthing looks ok. I see below error msg on HPA

-failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
-invalid metrics (1 invalid out of 1), first error is: failed to get cpu resource metric value: failed to get cpu utilization: did not receive metrics for targeted pods (pods might be unready)
-failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
-failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server is currently unable to handle the request (get pods.metrics.k8s.io)

Metrics server continue to print the same logs as before.

[dashpole]

/assign @CatherineF-dev @dgrisonnet
/triage accepted

[tengqm]

Connection to node hostname or IP from within the metrics-server Pod is a problem to me. I'm facing the same issue when using flannel.

[vgokul984]

@MahiraTechnology After opening the port 10250 and 443 on Node SG with the source range of VPC the issue fixed

[brankodjurkic]

Same as @vgokul984
I have containerPort: 4443 and hostNetwork: enabled in the values.yaml
Security Group on Node open 10250, both inbound and outbound solve an issue:

kubectl top nodes
ip-10-144-0-146.eu-west-1.compute.internal 45m 2% 2256Mi 31%
ip-10-144-0-17.eu-west-1.compute.internal 108m 5% 3244Mi 45%
ip-10-144-0-97.eu-west-1.compute.internal 78m 4% 3510Mi 49%