use signed kubelet serving certificates

[raravena80]

Is this a BUG REPORT or FEATURE REQUEST?

/kind bug

Opening the kubeadm side for this issue on the metrics-server

Versions

$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:43:08Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:46:06Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.1", GitCommit:"4ed3216f3ec431b140b1d899130a69fc671678f4", GitTreeState:"clean", BuildDate:"2018-10-05T16:36:14Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
  Any
• OS (e.g. from /etc/os-release):
  Any
• Kernel (e.g. uname -a):

$ uname -a
Linux ip-172-31-1-118 4.15.0-1023-aws #23-Ubuntu SMP Mon Sep 24 16:31:06 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

• Others:

What happened?

kubeadm creates certs under /var/lib/kubelet/pki/kubelet.* signed with a different CA from the one under /etc/kubernetes/pki/ca.pem

What you expected to happen?

As a result some apps like the metrics-server cannot collect stats from a secured kubelet because the kubelet has certs signed by a different ca from the K8s master(s)

Error sample:

E1108 23:49:32.090084       1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-x-x-x-x: unable to fetch metrics from Kubelet ip-x-x-x-x (ip-x-x-x-x): Get https://ip-x-x-x-x:10250/stats/summary/: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:ip-x-x-x-x: unable to fetch metrics from Kubelet ip-x-x-x-x (ip-x-x-x-x): Get https://ip-x-x-x-x:10250/stats/summary/: x509: certificate is valid for x.x.x.x not ip-x-x-x-x]

How to reproduce it (as minimally and precisely as possible)?

Install the metrics-server on run:

$ kubectl -n kube-system logs

Anything else we need to know?

Some more background here

There also steps in there that I followed to fix the issue.

---

edit: neolit123

the problem here is that the serving cert is self-signed by default:
see kubernetes/website#27071 for documentation update.

[fabriziopandini]

@raravena80 I'm not aware of any certificate created by kubeadm under /var/lib/kubelet/pki/.. could you kindly provide more info? e.g kubeadm config files, steps to create the cluster

[raravena80]

@fabriziopandini I'm not entirely sure if the certs are created by the kubeadm, but the general procedure is described here.

This is what the content of the directory looks like:

root@ip-172-31-1-118:/var/lib/kubelet/pki# pwd
/var/lib/kubelet/pki
root@ip-172-31-1-118:/var/lib/kubelet/pki# ls -al
total 24
drwxr-xr-x 2 root root 4096 Jul 23 21:10 .
drwxr-xr-x 7 root root 4096 Nov 12 04:52 ..
-rw------- 1 root root 2810 Jul 23 21:09 kubelet-client-2018-07-23-21-09-53.pem
-rw------- 1 root root 1159 Jul 23 21:10 kubelet-client-2018-07-23-21-10-43.pem
lrwxrwxrwx 1 root root   59 Jul 23 21:10 kubelet-client-current.pem -> /var/lib/kubelet/pki/kubelet-client-2018-07-23-21-10-43.pem
-rw-r--r-- 1 root root 1501 Nov  8 23:53 kubelet.crt
-rw------- 1 root root 1679 Nov  8 23:53 kubelet.key
root@ip-172-31-1-118:/var/lib/kubelet/pki#

Are they kubelet.crt and kubelet.key files created by the kubelet the first time it loads?

[fabriziopandini]

@raravena80 thanks for the clarification
Probably I don't have the full context here so I leave room to others for answering.

Only one side note (might be it can help)
Kubeadm already creates a certificate named apiserver-kubelet-client for letting the api server talking securely with the kubelets; it is signed by ca and bound to necessary RBAC rules.

[timothysc]

/assign @liztio

[anitgandhi]

I think this is to pre-generate kubelet's server certs. I tinkered with trying to use the Kubelet flags for TLS server bootstrap and rotate server certs, unfortunately I could not get Kubelet to request a server cert for itself using the bootstrap token. Kubelet ends up falling back to its default behavior for server certs, which is to generate a self-signed one.

To the best of my knowledge, right now the only way around that is to generate Kubelet's server certs out-of-band and place them at a deterministic path and kubelet (configured by kubeadm) will pick it up, and set some kubelet flags accordingly; reference: https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-tls-bootstrapping/#client-and-serving-certificates

The apiserver-kubelet-client is the client cert that API server will present to a kubelet, but kubelet is configured to trust clients that are signed by the k8s CA:

# cat /var/lib/kubelet/config.yaml 
address: 0.0.0.0
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 2m0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt

It's the kubelet's identity as a server being presented that needs to be signed by the k8s CA, which comes around to the original question.

There's also some relevant discussion at the end of this thread: #118

I think kubeadm may have to add a CSR approver for server cert requests with a valid bootstrap token, just like it does for client cert requests?

[kfox1111]

What about having the kubelet upload its self signed ca to a configmap somewhere? the nodeadmission plugin could restrict it to just its own configmap. metrics-server can use it to contact the node.

[stgleb]

Any ideas about that?

[alexbrand]

If possible, I think we should use the TLS bootstrapping facilities built into the kubelet for requesting/rotating serving certificates.

[stgleb]

@alexbrand I agree on that

[brokenjacobs]

kubelet TLS bootstrapping only generates client certificates for whatever reason:
--bootstrap-kubeconfig string
Path to a kubeconfig file that will be used to get client certificate for kubelet. If the file specified by --kubeconfig does not exist, the bootstrap kubeconfig is used to request a client certificate from the API server. On success, a kubeconfig file referencing the generated client certificate and key is written to the path specified by --kubeconfig. The client certificate and key file will be stored in the directory pointed by --cert-dir.

And kubeadm already does this. Perhaps this is a kubelet feature request?

[allir]

Running into this exact issue creating v1.18.2 cluster with kubeadm.

When setting up metrics-server it doesn't work without either setting it's kubelet-insecure-tls flag OR issue certificates for kublet "out of band", signing it with the kubernetes CA.

I thought about re-using the kubelet client certificate but that's of course issued to CN = system:node:nodename and no SANs. And I did test it though wich ofcourse changes the error to indicate just that. The same certificate could be used as both server/client if it had the nodename as a subject alternate name? But I'm guessing it'd be more proper to use separate certs for server/client?

[raravena80]

/remove-lifecycle frozen

[neolit123]

/lifecycle frozen

it's frozen so that the bots don't close the issue.

[neolit123]

The same certificate could be used as both server/client if it had the nodename as a subject alternate name?

in theory and unless the kubelet validates them - e.g. "client cert must not have SANs".

But I'm guessing it'd be more proper to use separate certs for server/client?

it is the common practice to use them separate even in cases where it seems avoidable. it's unlikely that the kubelet/auth{z|n} maintainers would change this detail.

[allir]

Hey. Did a little more diggin. Kubelet configuration option serverTLSBootstrap: true can actually create a CSR for the serving certificate. But it leaves it unapproved. Which may be ok?

Setting both rotateCertificates: true and serverTLSBootsrap: true and then approving the CSR for the serving certificate seem like the easiest way to go here. The serving certificate that is reqested/issued is for O = system:nodes, CN = system:node:<nodename> with Subject Alternate Names for DNS: <nodename>, IP Address: <node IP address>

Should kubeadm at enable the serverTLSBootstrap configuration option at least so Approving the server certificate would be an easy thing to do? Or even kubeadm could do the approval too?

[nijave]

Hey. Did a little more diggin. Kubelet configuration option serverTLSBootstrap: true can actually create a CSR for the serving certificate. But it leaves it unapproved. Which may be ok?

Setting both rotateCertificates: true and serverTLSBootsrap: true and then approving the CSR for the serving certificate seem like the easiest way to go here. The serving certificate that is reqested/issued is for O = system:nodes, CN = system:node:<nodename> with Subject Alternate Names for DNS: <nodename>, IP Address: <node IP address>

Should kubeadm at enable the serverTLSBootstrap configuration option at least so Approving the server certificate would be an easy thing to do? Or even kubeadm could do the approval too?

Not sure the security implementations but you can combine serverTLSBootstrap with this operator to auto-approve the CSRs https://github.com/kontena/kubelet-rubber-stamp

[neolit123]

Should kubeadm at enable the serverTLSBootstrap configuration option at least so Approving the server certificate would be an easy thing to do? Or even kubeadm could do the approval too?

kubeadm cannot do the approval because kubeadm is not a daemon. it has to deploy a controller / operator that manages that for the user. maybe in the future.

the certificates API should go GA soon and hopefully we are going to have a better way to manage this in k8s. please watch:
kubernetes/enhancements#267
(yet unclear to me what we are going to end up with...)

we also have alternatives ideas. but if all this is trying to solve the metric-server issue, you might as well just use https://github.com/brancz/kube-rbac-proxy that can perform SAR on the MS requests to the kubelet . sadly this is not yet documented on our side:
#1602

[allir]

@neolit123 I at least started looking into it when trying to stand up metrics server on kubeadm and "the hard way" clusters for learning experience. Easiest way there was of course to flag MS with --kubelet-insecure-tls, but I really wanted to see how to fix it in a secure way and then just got interested in the issue. 🙂

For now it's easy enough for me to add the serverTLSbootstrap flag to the kubelet config and manually approve the certificates. I have noticed a downside of it though, which is that you can't fully interact with pods on the node until you approve the cert. (kubectl exec fails to run command on pods running on a node before approval for example)

I'll follow along the enhancements issue too. Thanks.

[vainkop]

That's really sad that with kubeadm which seems mature enough the out of the box result for kubeletet cert is to be self signed & many people choose kubelet-insecure-tls for metrics server instead of doing things properly & etc :(

[neolit123]

it's a complicated problem.

please try:
https://github.com/kontena/kubelet-rubber-stamp
or
https://github.com/brancz/kube-rbac-proxy
as workarounds

[vainkop]

it's a complicated problem.

please try:
https://github.com/kontena/kubelet-rubber-stamp
or
https://github.com/brancz/kube-rbac-proxy
as workarounds

Actually https://github.com/kontena/kubelet-rubber-stamp works pretty good & imo seems to be a more correct solution instead of proxy.

Step 1:
Add
serverTLSBootstrap: true to the end of every /var/lib/kubelet/config.yaml for kubelets reconfiguration & don't forget to apply config (or just reboot them)

Step 2:
Deploy kubelet-rubber-stamp

service_account.yaml
role.yaml
role_binding.yaml
operator.yaml

Step 3:
Edit metrics-server deployment & remove --kubelet-insecure-tls

Result:

kubectl get csr
NAME        AGE   SIGNERNAME                      REQUESTOR          CONDITION
csr-7dvsx   31m   kubernetes.io/kubelet-serving   system:node:u-02   Approved,Issued
csr-d6rvm   31m   kubernetes.io/kubelet-serving   system:node:u-03   Approved,Issued
csr-szblz   31m   kubernetes.io/kubelet-serving   system:node:u-01   Approved,Issued
csr-zjfgj   31m   kubernetes.io/kubelet-serving   system:node:u-04   Approved,Issued

[allir]

Hey, just to add to that @vainkop
During your initial kubeadm init to create the cluster you should also be able to pass in a KubeletConfiguration API object file to set the serverTLSBootstrap

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
...
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
serverTLSBootstrap: true

kubeadm init --config=kubeadm-config.yaml

Then all kubelet's will automatically be set up using the serverTLSBootstrap flag.

To get the CSRs

kubectl get csr
NAME        AGE     SIGNERNAME                                    REQUESTOR                 CONDITION
csr-2qkdw   2m1s    kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:fcufbo   Approved,Issued
csr-9wvgt   114s    kubernetes.io/kubelet-serving                 system:node:worker-1      Pending
csr-lz97v   4m58s   kubernetes.io/kubelet-serving                 system:node:master-1      Pending
csr-rsdsp   4m59s   kubernetes.io/kube-apiserver-client-kubelet   system:node:master-1      Approved,Issued
csr-wgxqs   4m49s   kubernetes.io/kubelet-serving                 system:node:master-1      Pending

Then either approve them manually or deploy https://github.com/kontena/kubelet-rubber-stamp which approves them automatically. I just tried it with kubelet-rubber-stamp and it works great.

Also I did not seem to need to restart the kubelet's this way, they picked up their certificates as soon as I approvde the CSR, but a caveat is that the kublet's have NO cert until the CSR is approved, it does not get a self signed certificate first.

kubectl certificate approve csr-ab123           # OR deploy rubber-stamp!

kubectl get csr
NAME        AGE   SIGNERNAME                      REQUESTOR              CONDITION
csr-9wvgt   3m    kubernetes.io/kubelet-serving                 system:node:worker-1      Approved,Issued
...

Another strange thing seems to happen here btw, which is that the master node seems to create it’s CSR twice. (At least the two times I tried this)

But as @nijave says in a comment above, I'm not sure what the security implications of using the rubber-stamper are.

[equinox0815]

@allir, @vainkop as far is i can see the kubelet-rubber-stamp does only verify if the common name of the CSR matches the requester name but does not verify whether the additional hostnames and IP Addresses requested by the kubelet are valid. This means an attacker that has access to the kubelet client certificate can create certificates for basically any domain name or IP address. All clients that are configured to trust the root CA will then accept this certificate.
Of course validating the hostname and IP addresses that are valid for a given kubelet is hard since there is currently no authority that can confirm what a kubelet is allowed to request. For example using the node object on the API server is not sufficient because kubelets can update the object without limits.

[vainkop]

Hey, just to add to that @vainkop
During your initial kubeadm init to create the cluster you should also be able to pass in a KubeletConfiguration API object file to set the serverTLSBootstrap
kubeadm init --config=kubeadm-config.yaml
Then all kubelet's will automatically be set up using the serverTLSBootstrap flag.

Or for an existing K8s setup using Ansible it can be:

  tasks:
    - name: Insert a line at the end of /var/lib/kubelet/config.yaml
      lineinfile:
        path: /var/lib/kubelet/config.yaml
        line: 'serverTLSBootstrap: true'

+restart kubelets

[kvaps]

Wow I'm so glad that I found this issue, and I'm not alone who wants to make this proper way. :)

Now let me share my thoughts on this issue (please correct me if I'm somewhere wrong):

First my vision of the original problem:
Currently kubeadm enables webhook authentication for all kubelets by default, so kubelet is validating client certificates for incoming connections with no problem even if --kubelet-insecure-tls option is specified.
From the other side metrics-server have no oportunity to verify specific kubelet certificate because it is self-signed on the node.

Possible risks of using --kubelet-insecure-tls for metrics-server:
While the kubelet data is somewhat secured and will never be provided to the metrics-server without successful webhook authentification.
In theory someone can compromise the server IP or hostname and provide wrong statistics. But for establishing the connections metricserver is using an IP address and hostnames specified for node via kube-apiserver, so the attacker needs to hack the API-server, DNS or the node IP-address first.

Little observation:
The metrics-server is not single service which is accessing the kubelets directly. Kube-apiserver also doing this for read the container logs or execute the shell on them. The good question is how kube-apiserver makes sure that it establishing connection with the specific kubelet while it has no any information about CA that issued it's certificate?
Doesn't it behave the same as the metrics server with the --kubelet-insecure-tls option in this case?

Possible solution:
Nowdays the webhooks and API aggregation are quite popular in Kubernetes. All of them are behave the similar way, by generating it's own CA and crt/key pair. The CA hash is also stored in a specific resource to provide kube-apiserver with information about which certificate it can trust.

For example:

• APIServices are storing related CA hash in their apiservices.apiregistration.k8s.io resource:

  spec:
    caBundle: <ca-hash>

• Webhooks are storing related CA hash in their validatingwebhookconfigurations.admissionregistration.k8s.io and mutatingwebhookconfigurations.admissionregistration.k8s.io resources:

  webhooks:
  - clientConfig:
     caBundle: <ca-hash>

To me it is pretty obvious that each node resource should have similar caBundle in their spec, where kubelets can register their own CA for serving using their client certificate:

spec:
  caBundle: <ca-hash>

Both metris-server and kube-apiserver should use these certificates to verify and trust the connection to the kubelets.

thanks to @kfox1111 who expressed a similar idea previously #1223 (comment)

[kvaps]

The good question is how kube-apiserver makes sure that it establishing connection with the specific kubelet while it has no any information about CA that issued it's certificate?
Doesn't it behave the same as the metrics server with the --kubelet-insecure-tls option in this case?

To answer this question I can quote @luxas here:

Right, we can't make the connections from the api server to the kubelet servers verified, as each kubelet has its own self-signed cert. We might consider a manual approving flow wrt kubelet serving certs in the future but that's not secured by default at the moment.

from #118 (comment)

[willzhang]

hope it can be resolved some day

[root@jenkins metrics-server]# kubectl -n kube-system logs -f metrics-server-6955d88db9-lftlz
I1120 08:23:09.094132       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I1120 08:23:09.094234       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I1120 08:23:09.094270       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1120 08:23:09.094279       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1120 08:23:09.094307       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1120 08:23:09.094315       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1120 08:23:09.095064       1 dynamic_serving_content.go:130] Starting serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key
I1120 08:23:09.095207       1 secure_serving.go:197] Serving securely on [::]:4443
I1120 08:23:09.095259       1 tlsconfig.go:240] Starting DynamicServingCertificateController
I1120 08:23:09.194453       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file 
I1120 08:23:09.194660       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file 
I1120 08:23:09.194455       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController 
E1120 08:23:10.420643       1 server.go:132] unable to fully scrape metrics: [unable to fully scrape metrics from node k8s-master3: unable to fetch metrics from node k8s-master3: Get "https://10.39.140.250:10250/stats/summary?only_cpu_and_memory=true": x509: cannot validate certificate for 10.39.140.250 because it doesn't contain any IP SANs, unable to fully scrape metrics from node k8s-master1: unable to fetch metrics from node k8s-master1: Get "https://10.39.140.248:10250/stats/summary?only_cpu_and_memory=true": x509: cannot validate certificate for 10.39.140.248 because it doesn't contain any IP SANs, unable to fully scrape metrics from node k8s-master2: unable to fetch metrics from node k8s-master2: Get "https://10.39.140.249:10250/stats/summary?only_cpu_and_memory=true": x509: cannot validate certificate for 10.39.140.249 because it doesn't contain any IP SANs, unable to fully scrape metrics from node k8s-node1: unable to fetch metrics from node k8s-node1: Get "https://10.39.140.251:10250/stats/summary?only_cpu_and_memory=true": x509: cannot validate certificate for 10.39.140.251 because it doesn't contain any IP SANs]
I1120 08:23:33.874949       1 requestheader_controller.go:183] Shutting down RequestHeaderAuthRequestController
I1120 08:23:33.874978       1 configmap_cafile_content.go:223] Shutting down client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1120 08:23:33.874993       1 configmap_cafile_content.go:223] Shutting down client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1120 08:23:33.875019       1 tlsconfig.go:255] Shutting down DynamicServingCertificateController
I1120 08:23:33.875026       1 dynamic_serving_content.go:145] Shutting down serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key
I1120 08:23:33.875041       1 secure_serving.go:241] Stopped listening on [::]:4443

[maxclac]

No news from this issue? I would be interested to have a solution for this as well.

[neolit123]

we are documenting workarounds here:
kubernetes/website#27071
#1602

we can keep this issue open, but due to the complexities of requiring to deploy a signer with kubeadm by default, it's unlikely that we are going to make this change anytime soon.

[neolit123]

This would mean keeping track of the previous state. Maybe it can work. You can talk with the maintainers about this idea in #sig-auth on k8s slack.

…

On Oct 6, 2021 12:08, "DrmagicE" ***@***.***> wrote: How about make the kube-controller can auto-approve those CSRs that are not changing anything or not changing some sensitive field such as IP address, DNS name, common name, organization etc.? I think in most cases, we just want to renew the certificate without any changes. It would be nice if the kube-controller can auto-approve those CSRs. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1223 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACRATDGFQUXQWCV4XE2HIDUFQGZPANCNFSM4GCXG3JQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[ViliusS]

we are documenting workarounds here: kubernetes/website#27071 #1602

we can keep this issue open, but due to the complexities of requiring to deploy a signer with kubeadm by default, it's unlikely that we are going to make this change anytime soon.

Can someone comment on what are possible consequences if we would use serverTLSBootstrap: true configuration as a workaround? Does it enable some other processes we should know about? Or better yet, why is this setting not enabled by default?

It would help us to understand possible risks and decide if enabling this configuration is somewhat better than running metrics-server with --kubelet-insecure-tls (we basically need it only for metrics-server).

[kfox1111]

Some kind of integration with spire would be interesting. There is a systemd workload attestor now, so targeting a cert to a kubelet via systemd kubelet service would be much easier to do now.