/var/lib/kubelet/pki/kubelet.crt not renewed with kubeadm alpha certs renew all

[rdxmb]

BUG REPORT

Versions

kubeadm version (use kubeadm version):
kubeadm version: &version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:12:12Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):
  kubectl version

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:14:22Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:07:13Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
  Hardware

• OS (e.g. from /etc/os-release):
  Ubuntu 16.04..6 LTS

• Kernel (e.g. uname -a):
  Linux eins 4.15.0-88-generic #88~16.04.1-Ubuntu SMP Wed Feb 12 04:19:15 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

• Others:

What happened?

When running kubeadm alpha certs renew all, the cert in /var/lib/kubelet/pki/kubelet.crt is not renewed. See the monitoring screenshot below.

However, I am not sure if this file is really used in the cluster.

If this is no bug and work as designed, I suggest to delete the file or at least have a documentation, which certs (in which path) are used.

kubeadm alpha certs renew all
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

Maybe this is a side effect of #1223

[neolit123]

hello,

please see https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/

Note: kubelet.conf is not included in the list above because kubeadm configures kubelet for automatic certificate renewal.

this certificate is rotated out of band and automatically by the kubelet process. i believe it should happen around 80% of the total cert validity duration.

also see this issue that we closed a couple of days ago:
#2185

i'm going to close this but let me know if you have further questions.

/close
/triage support

[k8s-ci-robot]

@neolit123: Closing this issue.

In response to this:

hello,

please see https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/

Note: kubelet.conf is not included in the list above because kubeadm configures kubelet for automatic certificate renewal.

this certificate is rotated out of band and automatically by the kubelet process. i believe it should happen around 80% of the total cert validity duration.

also see this issue that we closed a couple of days ago:
#2185

i'm going to close this but let me know if you have further questions.

/close
/triage support

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[rdxmb]

@neolit123 Thanks for your answer!

i'm going to close this but let me know if you have further questions.

Does that mean, if I changed the kubelet.conf like described in the docs, /var/lib/kubelet/pki/kubelet.crt and /var/lib/kubelet/pki/kubelet.key can be deleted?

[neolit123]

Does that mean, if I changed the kubelet.conf like described in the docs, /var/lib/kubelet/pki/kubelet.crt and /var/lib/kubelet/pki/kubelet.key can be deleted?

that is not what the docs describe.
these files are the kubelet serving certificate and key and they don't apply to kubelet.conf

depends on what you want to change, but the docs instruct you to link your kubelet.conf to the kubelet-client-current.pem file in that folder.

https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/

[rdxmb]

yes, I got it. Just put the kubelet-client-current.pem to my monitoring and removed the others from the monitoring. I guess they are obsolete.

The link in kubelet.conf is added since the changelog described that ;)

Thanks again.

[blepoutr]

Hello

In my opinion, /var/lib/kubelet/pki/kubelet.crt is not an obsolete file. It seems as a bundle PEM that contains Self signed CA and associated Server certificate that is used by kubelet to serve "contents". It is typically used by metrics-server for example.

It is actually a real problem that kubelet uses self signed certificates to serve "metrics" as it enforces the usage of --kubelet-insecure-tls in metrics-server deployment.

see https://github.com/kubernetes/kubeadm/issues/1223

[pmmalinov01]

@blepoutr Thank for explaining about /var/lib/kubelet/pki/kubelet.crt, Can you just verify that the kubelet.crt is not used by the kubelet to talk to the API server, but used when metrics are served? Is there documentation?

[blepoutr]

@pmmalinov01,
I don't know if api server is calling kubelet service ( and so is also concerned by checking the self signed certificate serving the kubelet api content). What I know is that I had the bug when metrics-server tries to consume kubelet Api content.

I solved my issue, by forcing kubelet not to use selfsigned certificate as Serving Certificate (as per doc : https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/#client-and-serving-certificates)

Indeed, I have activated the RotateKubeletServerCertificate=true in order to have kubelet requesting a Serving certificate to K8S itself. Besides, it will automatically do it, when a new node is installed, or if the previous existing Serving certificate is going to expire.

The good news, is that it allows me to have CA check between Metrics server and kubelet and so I can stop using --kubelet-insecure-tls.

The only drawback is that I (as K8S adminstrator) needs to approve the CSR manually ( by kubectl csr approve ...), before the certificate is generated and retrieved by kubelet. I amy consider using such tools to overcome that : https://github.com/kontena/kubelet-rubber-stamp

[andy-ning]

I feel like people are talking about two different certificates in /var/lib/kubelet/pki directory. In my 1.18 k8s system I have the following in that directory:

ll /var/lib/kubelet/pki/
total 12
-rw------- 1 root root 2765 Jan 19 19:51 kubelet-client-2021-01-19-19-51-34.pem
lrwxrwxrwx 1 root root 59 Jan 19 19:51 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2021-01-19-19-51-34.pem
-rw-r--r-- 1 root root 2193 Jan 19 19:51 kubelet.crt
-rw------- 1 root root 1675 Jan 19 19:51 kubelet.key

The kubelet-client-current.pem is what kubelet.conf pointing to. But I just couldn't find anybody else is using kubelet.crt/key (it's self signed non-CA cert). Maybe they are used initially and then later on rotated to use the kubelet-client-current.pem?

[neolit123]

kubelet.crt is the kubelet serving cert and it's self-signed.

it's not managed by kubeadm:
#1223
#1602

[andy-ning]

Dig a little bit more ... kubelet-client-current.pem contains a client cert, seems to be used for kubelet to securely connect to apiserver etc. kubelet.crt is a server cert, seems to be for components that try to securely connect to kubelet to verify kubelet's identity. So they are for different purposes. In my system, there seems to be no components are talking to kubelet (only kubelet talks to other components). That's probably why I don't find other components are using kubelet.crt/key.

[andy-ning]

kubelet.crt is the kubelet serving cert and it's self-signed.

it's not managed by kubeadm:
#1223
#1602
Thanks. So kubelet.crt/key is the one that will be automatically rotated by kubelet if configured so? What about kubelet-client-current.pem? Is it managed by kubeadm and will be renewed by "kubectl certs renew"?

[neolit123]

both the kublet serving and client certs can be auto-rotated.
the client cert rotation is currently enabled by default in 1.20.
the serving cert is not and the above issues try to explain why.

for more info:
https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/#client-and-serving-certificates

[shaktiprasadrath]

I have similar problem. My kubelet.crt is expired. i don't know a suitable way to renew it.
When I am trying the below I am getting error:

kl get no
error: You must be logged in to the server (Unauthorized)
Note: kl is alias of kubectl

Is there a way to renew the kubelet.crt/key?

I read above conversation but still didn't get any concrete answer to renew my kubelet.crt/key

[FedericoHeichou]

I have similar problem. My kubelet.crt is expired. i don't know a suitable way to renew it. When I am trying the below I am getting error:

kl get no error: You must be logged in to the server (Unauthorized) Note: kl is alias of kubectl

Is there a way to renew the kubelet.crt/key?

I read above conversation but still didn't get any concrete answer to renew my kubelet.crt/key

Did you found a solution? I still have this problem

[neolit123]

there is a guide for recovering the kubelet failed state due to outdated certs here:
https://github.com/kubernetes/website/blob/6ea53189723803c363d4de1c0cc76dd0cd98ff39/content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md#kubelet-client-certificate-rotation-fails-kubelet-client-cert

the documentation there is currently being edited and will be available here:
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm/#kubelet-client-cert

[akshaysharama]

there is a guide for recovering the kubelet failed state due to outdated certs here: https://github.com/kubernetes/website/blob/6ea53189723803c363d4de1c0cc76dd0cd98ff39/content/en/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm.md#kubelet-client-certificate-rotation-fails-kubelet-client-cert

the documentation there is currently being edited and will be available here: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/troubleshooting-kubeadm/#kubelet-client-cert

this is not working, removed kubelet.conf and certs from /var/lib/kubelet/pki/*, and replace with kubelet.conf generated as mentioned in above link and restarted the kubelet service, but still it is failing and not creating the certs.

[FedericoHeichou]

Anyway I resolved with (in each node)

# Here you will see the cert not renewed
openssl s_client -connect 127.0.0.1:10250  | grep NotBefore

# Add the rotation of server certs (check if there is no conf before adding it)
# https://kubernetes.io/docs/reference/access-authn-authz/kubelet-tls-bootstrapping/#certificate-rotation
echo "serverTLSBootstrap: true" >> /etc/kubernetes/kubelet-config.yaml
systemctl restart kubelet

# csr are created after restart, you need to manually approve them
kubectl get csr
kubectl certificate approve csr-xxxxx

# Here you will see the cert is renewed
openssl s_client -connect 127.0.0.1:10250 | grep NotBefore