Failed to scrape node: remote error: tls: internal error

[rarecrumb]

What happened: Metrics server failed to scrape a node

What you expected to happen: Successfully scrape the node

Anything else we need to know?: Deploying with the helm chart

Environment:

• Kubernetes distribution (GKE, EKS, Kubeadm, the hard way, etc.): EKS

• Container Network Setup (flannel, calico, etc.): Calico

• Kubernetes version (use kubectl version): 1.29

• Metrics Server manifest

spoiler for Metrics Server manifest:

      args:
      - --kubelet-insecure-tls
      containerPort: 4443
      hostNetwork:
        enabled: true

• Kubelet config:

spoiler for Kubelet config:

• Metrics server logs:

spoiler for Metrics Server logs:

E0501 16:40:35.362224       1 scraper.go:149] "Failed to scrape node" err="Get \"https://10.3.10.48:10250/metrics/resource\": remote error: tls: internal error" node="ip-10-3-10-48.ec2.internal"

• Status of Metrics API:

spolier for Status of Metrics API:

kubectl describe apiservice v1beta1.metrics.k8s.io

Name:         v1beta1.metrics.k8s.io
Namespace:
Labels:       app.kubernetes.io/instance=metrics-server
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=metrics-server
              app.kubernetes.io/version=0.7.1
              argocd.argoproj.io/instance=metrics-server
              helm.sh/chart=metrics-server-3.12.1
Annotations:  <none>
API Version:  apiregistration.k8s.io/v1
Kind:         APIService
Metadata:
  Creation Timestamp:  2023-07-13T20:41:49Z
  Resource Version:    266080474
  UID:                 59bdff53-5db0-4819-a27e-6aff8526d41e
Spec:
  Group:                     metrics.k8s.io
  Group Priority Minimum:    100
  Insecure Skip TLS Verify:  true
  Service:
    Name:            metrics-server
    Namespace:       base
    Port:            443
  Version:           v1beta1
  Version Priority:  100
Status:
  Conditions:
    Last Transition Time:  2024-04-30T18:18:43Z
    Message:               all checks passed
    Reason:                Passed
    Status:                True
    Type:                  Available
Events:                    <none>

/kind bug

[logicalhan]

/kind support
/triage accepted