After renew the certs getting error in kube-apiserver pod logs : Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid

[sagarputta]

What happened?

Certificates renewed with using this command: kubeadm alpha certs renew all
Kubernetes version: 1.19.0
My issue is after renewed kubernetes certificates
I'm getting x509 certificate has expired error in kube-apiserver pod logs
Restarted docker containerd & kubelet but still getting same issue in kube-apiserver pod logs
kubectl logs kube-apiserver -n kube-system
E0925 22:45:43.610253 1 authentication.go:53] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid: current time 2023-09-25T22:45:43Z is after 2023-09-25T12:10:16Z I0925 22:45:44.656413 1 client.go:360] parsed scheme: "passthrough" I0925 22:45:44.656478 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://127.0.0.1:2379/ 0 }] } I0925 22:45:44.656494 1 clientconn.go:948] ClientConn switching balancer to "pick_first" E0925 22:45:45.104633 1 authentication.go:53] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid: current time 2023-09-25T22:45:45Z is after 2023-09-25T12:10:15Z E0925 22:45:46.208945 1 authentication.go:53] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid: current time 2023-09-25T22:45:46Z is after 2023-09-25T12:10:16Z

What did you expect to happen?

its as to show apiserver logs as usual but its showing
"E0926 05:27:20.735229 1 authentication.go:53] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid: current time 2023-09-26T05:27:20Z is after 2023-09-25T12:10:16Z
I0926 05:27:21.802752 1 client.go:360] parsed scheme: "passthrough"
I0926 05:27:21.802824 1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://127.0.0.1:2379 0 }] }
I0926 05:27:21.802835 1 clientconn.go:948] ClientConn switching balancer to "pick_first"
E0926 05:27:22.129331 1 authentication.go:53] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid: current time 2023-09-26T05:27:22Z is after 2023-09-25T12:10:16Z
E0926 05:27:22.765121 1 authentication.go:53] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid: current time 2023-09-26T05:27:22Z is after 2023-09-25T12:10:15Z
E0926 05:27:22.900270 1 authentication.go:53] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid: current time 2023-09-26T05:27:22Z is after 2023-09-25T12:10:16Z
E0926 05:27:24.420903 1 authentication.go:53] Unable to authenticate the request due to an error: invalid bearer token
E0926 05:27:24.914140 1 authentication.go:53] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid: current time 2023-09-26T05:27:24Z is after 2023-09-25T12:10:16Z
E0926 05:27:25.070424 1 authentication.go:53] Unable to authenticate the request due to an error: invalid bearer token
E0926 05:27:25.495518 1 authentication.go:53] Unable to authenticate the request due to an error: x509: certificate has expired or is not yet valid: current time 2023-09-26T05:27:25Z is after 2023-09-25T12:10:15Z
E0926 05:27:26.504174 1 authentication.go:53] Unable to authenticate the request due to an error: invalid bearer token"

How can we reproduce it (as minimally and precisely as possible)?

renew the certs and restart docker kubelet and check kube-apiserver pod logs

Anything else we need to know?

No response

Kubernetes version

single master cluster
kubernetes version v1.19.0
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"e19964183377d0ec2052d1f1fa930c4d7575bd50", GitTreeState:"clean", BuildDate:"2020-08-26T14:30:33Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"e19964183377d0ec2052d1f1fa930c4d7575bd50", GitTreeState:"clean", BuildDate:"2020-08-26T14:23:04Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider

On-premises (private cloud)

OS version

# On Linux:
$ cat /etc/os-release
# NAME="Red Hat Enterprise Linux Server"
VERSION="7.9 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.9"
PRETTY_NAME="Red Hat"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.9:GA:server"
$ uname -a
# Linux servername 3.10.0-1160.76.1.el7.x86_64 #1 SMP Tue Jul 26 14:15:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux




### Install tools

docker
containerd 
kubernetes

### Container runtime (CRI) and version (if applicable)


Server: Docker Engine - Enterprise
 Engine:
  Version:          19.03.8

### Related plugins (CNI, CSI, ...) and versions (if applicable)

calico version v3.13.3

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[tamilselvan1102]

/sig network

[tamilselvan1102]

this topic is discussed here: #581

[tamilselvan1102]

For Certificate Management with kubeadm refer below
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/

Check certificate expiration
kubeadm alpha certs check-expiration

[neolit123]

probably a setup problem.

a better place to ask would be on the support channels. please see:
https://git.k8s.io/kubernetes/SUPPORT.md

/kind support
/close

[k8s-ci-robot]

@neolit123: Closing this issue.

In response to this:

probably a setup problem.

a better place to ask would be on the support channels. please see:
https://git.k8s.io/kubernetes/SUPPORT.md

/kind support
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.