enable kubelet server to dynamically load tls certificate files

[zhangweikop]

What type of PR is this?

/kind bug
/kind feature

What this PR does / why we need it:

It enable kubelet tls server to watch and reload certificate files.

In our kubelet deployment, we don't use ServerTLSBootstrap or let kubelet to create a self-signed cert.
We have other mechanism (based on internal PKI & CA) to generate and rotate the certificate files for kubelet on each host.

Therefore, we want that kubelet TLS (both client and server side) can handle the certificate update. This feature is important for us when we start generating short shorter-lived certificates for kubelet

Currently in Kubelet:

• The client side TLS has been handling the certificate reload very well thanks to the client-go.
  In our testing, the kubelet -> API server requests never encountered issues. The pod can be deployed and started correctly on the host after the kubelet certificate file expired and then regenerated.
• But the server side TLS doesn't have auto-reload. Therefore, the https requests to kubelet (such as access container kubectl log or kubectl exec) would fail with following error message if the certificate loaded in kubelet server TLSconfig expired.

Error from server: Get "https://<...hostname>.com:20250/containerLogs/....: x509: certificate has expired or is not yet valid: current time 2024-04-24T00:11:34Z is after 2024-04-12T00:52:35Z

Special notes for your reviewer:

1. The logic is expressed as a certificate.Manager so that it could be used similarly to other struct in pkg/kubelet/certificate/kubelet.go.
2. The internal file watch and read implementation is based on DynamicCertKeyPairContent struct in k8s.io/apiserver/pkg/server/dynamiccertificates.

Does this PR introduce a user-facing change?

kubelet server can now dynamically load certificate files

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: zhangweikop / name: wzhang (f38499c, 2aba380, da7c8e6, 44746e4, a4a3e72, e740a83)

[k8s-ci-robot]

Welcome @zhangweikop!

It looks like this is your first PR to kubernetes/kubernetes 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kubernetes has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @zhangweikop. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[kannon92]

/ok-to-test

[kannon92]

I would suggest taking this feature to sig-auth.

[kannon92]

/remove-kind bug

[bart0sh]

/triage accepted
/priority important-longterm

[enj]

2. The internal file watch and read implementation is based on DynamicCertKeyPairContent struct in k8s.io/apiserver/pkg/server/dynamiccertificates.

I would expect that logic to be re-used, not copied.

[enj]

2. The internal file watch and read implementation is based on DynamicCertKeyPairContent struct in k8s.io/apiserver/pkg/server/dynamiccertificates.

I would expect that logic to be re-used, not copied.

Sorry, ignore me, I read the description without reading the code. I see that the code is re-used.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: zhangweikop
Once this PR has been reviewed and has the lgtm label, please assign derekwaynecarr for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• pkg/features/OWNERS
• pkg/kubelet/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[enj]

Minor comments.

This LGTM from an auth perspective.

I will defer to SIG node to determine if they are happy with the code, and if this change requires a feature gate / if that gate should be enabled or disabled by default.