certificate expiration after 1 day

[bilsch-nice]

I've been trying to understand why this is happening. The certificate generated by istio appears to be set for a validity of 1 day and then not getting renewed.

I'm using istioctl to install istio. The version is 1.21.0, kubernetes version is 1.29 ( kind inside docker )

The application in question is grafana tempo. The cluster component fails to join the cluster for all instances. I enabled debug logging of envoy and noticed something interesting:

2024-04-15T18:06:11.766360Z     debug   envoy connection external/envoy/source/extensions/transport_sockets/tls/cert_validator/default_validator.cc:325 verify cert failed: X509_verify_cert: certificate verification error at depth 0: certificate has expired      thread=26

When I uninstalled istio, removed the ca cert secret from the namespace and re-installed ( restarting all pods in-between ) istio everything lights up again. When I looked at the certificate generated:

( This certificate is still valid at the time of the post - showing for demonstration purposes )

❯ istioctl proxy-config secret -n tempo tempo-distributor-779bf4b8b5-s7kpp
RESOURCE NAME     TYPE           STATUS     VALID CERT     SERIAL NUMBER                        NOT AFTER                NOT BEFORE
default           Cert Chain     ACTIVE     true           d0b6aa3d29036c009514137deb27b317     2024-04-17T14:10:12Z     2024-04-16T14:08:12Z
ROOTCA            CA             ACTIVE     true           5a16725f5d4a1a1a7b90fb1227db3484     2034-04-07T18:32:21Z     2024-04-09T18:32:21Z

Shouldn't this certificate be renewed automatically? Is there a configuration which is not set by default that should be? I'm not using cert-manager for certificates as I expected istio and its components to handle this renewal for me.

I've done a lot of google searching and not come up with anything related to cert renewal so coming here hoping for some help ;)

[bilsch-nice]

I'm not sure if this is related but it sure seems like it would be

#38081

[bilsch-nice]

Is it possible there are configs that need to be tweaked to enable certificate renewal? What component does the renewal is it istio or envoy making the cert renewal request?

❯ ./istioctl proxy-config secret -n tempo tempo-distributor-778cd69c46-g5gsf
RESOURCE NAME     TYPE           STATUS     VALID CERT     SERIAL NUMBER                        NOT AFTER                NOT BEFORE
default           Cert Chain     ACTIVE     false          d6fd3ab6255ff7b5459b9f43d758e75c     2024-04-18T18:41:36Z     2024-04-17T18:39:36Z
ROOTCA            CA             ACTIVE     true           5a16725f5d4a1a1a7b90fb1227db3484     2034-04-07T18:32:21Z     2024-04-09T18:32:21Z

[howardjohn]

Its the istio-agent Go component in the istio container (there are 2 binaries, not just envoy).

Are you sleeping your computer with kind overnight or something? I have found that often causes issues (even with k8s itself without Istio) around things expiring

[bilsch-nice]

I've experienced this issue both ways - leaving my laptop up/running for multiple days at a time and seeing the certs expire. I have yet to see the certificates renew successfully.

Is my istio installation missing something maybe?

❯ kubectl get all -n istio-system
NAME                          READY   STATUS    RESTARTS   AGE
pod/istiod-7c9cd6ddc8-6l2tl   1/1     Running   0          43h

NAME             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                 AGE
service/istiod   ClusterIP   10.96.204.176   <none>        15010/TCP,15012/TCP,443/TCP,15014/TCP   43h

NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/istiod   1/1     1            1           43h

NAME                                DESIRED   CURRENT   READY   AGE
replicaset.apps/istiod-7c9cd6ddc8   1         1         1       43h

NAME                                         REFERENCE           TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/istiod   Deployment/istiod   <unknown>/80%   1         5         1          43h

[bilsch-nice]

So I'm trying to collect additional information on this issue. I toggled debug logging in istiod for all of the components. I have a log -f running on one of the pods with proxy logging set to debug.

istiod logging:

istioctl admin log  | awk '{print $1}' | while read x; do istioctl admin log --level $x:debug; done

❯ ~/downloads/istioctl admin log
ACTIVE SCOPE           DESCRIPTION                                  LOG LEVEL
ads                    ads debugging                                debug
adsc                   adsc debugging                               debug
analysis               Scope for configuration analysis runtime     debug
authn                  authn debugging                              debug
authorization          Istio Authorization Policy                   debug
ca                     ca client                                    debug
controllers            common controller logic                      debug
default                Unscoped logging messages.                   debug
delta                  delta xds debugging                          debug
file                   File client messages                         debug
gateway                gateway-api controller                       debug
grpcgen                xDS Generator for Proxyless gRPC             debug
ingress status                                                      info
klog                                                                debug
kube                   Kubernetes client messages                   debug
model                  model                                        debug
monitor                file configuration monitor                   debug
monitoring             metrics monitoring                           debug
pkica                  Citadel CA log                               debug
pkira                  Istiod RA log                                debug
processing             Scope for configuration processing runtime   debug
retry                  logs for retries                             debug
rootcertrotator        Self-signed CA root cert rotator log         debug
secretcontroller       Citadel kubernetes controller log            debug
security               security debugging                           debug
serverca               Citadel server log                           debug
serviceentry           ServiceEntry registry                        debug
spiffe                 SPIFFE library logging                       debug
status                 status controller for istio                  debug
trustBundle            Workload mTLS trust bundle logs              debug
validation             CRD validation debugging                     debug
validationController   validation webhook controller                debug
validationServer       validation webhook server                    debug
wasm                                                                debug
wle                    wle controller debugging                     debug

For the pod I picked one of the pods and toggled debug logging for the proxy:

~/downloads/istioctl proxy-config log -n loki loki-loki-distributed-distributor-d9bd9b76f-wtt25 --level=debug

The cert:

❯ ~/downloads/istioctl proxy-config secret -n loki loki-loki-distributed-distributor-d9bd9b76f-wtt25 | tee -a loki.loki-loki-distributed-distributor-d9bd9b76f-wtt25.s
ecret
RESOURCE NAME     TYPE           STATUS     VALID CERT     SERIAL NUMBER                        NOT AFTER                NOT BEFORE
default           Cert Chain     ACTIVE     true           77b99f8d9d5fe9dce1ace57108ddb9fa     2024-04-23T13:20:01Z     2024-04-22T13:18:01Z
ROOTCA            CA             ACTIVE     true           5a16725f5d4a1a1a7b90fb1227db3484     2034-04-07T18:32:21Z     2024-04-09T18:32:21Z

I can see the certificate generated:

2024-04-22T13:20:01.278712Z     info    ads     CDS: PUSH request for node:loki-loki-distributed-distributor-d9bd9b76f-wtt25.loki resources:109 size:92.7kB cached:103/103 nonce:92ac13c1-f9cf-4ce5-9b4c-a14afa485251 version:2024-04-22T13:20:01Z/11
2024-04-22T13:20:01.322281Z     debug   security        Authentication successful through auth source 1
2024-04-22T13:20:01.322352Z     debug   serverca        generating a certificate for client 10.244.0.207:51460, sans: [spiffe://cluster.local/ns/loki/sa/loki-loki-distributed], requested ttl: 24h0m0s
2024-04-22T13:20:01.325266Z     debug   serverca        CSR successfully signed for client 10.244.0.207:51460, sans [spiffe://cluster.local/ns/loki/sa/loki-loki-distributed].

Looking in the istio-proxy pod I see 2 processes:

❯ kubectl exec -it -n loki loki-loki-distributed-distributor-d9bd9b76f-wtt25 -c istio-proxy -- sh
$ ps -eaf
UID          PID    PPID  C STIME TTY          TIME CMD
istio-p+       1       0  0 13:20 ?        00:00:01 /usr/local/bin/pilot-agent proxy sidecar --domain loki.svc.cluster.local --proxyLogLevel=warning --proxyComponentL
istio-p+      18       1  4 13:20 ?        00:00:27 /usr/local/bin/envoy -c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-addres
istio-p+      31       0  1 13:30 pts/0    00:00:00 sh
istio-p+      37      31  0 13:30 pts/0    00:00:00 ps -eaf

I can't think of anything else to capture - please let me know if there is more I can grab!
Now to wait and see when the cert expires. I'm curious if this is somehow an issue in envoy-proxy not requesting the renewal. I found the istiod log for the cert generation so we should be able to prove that with the logging I've toggled

[bilsch-nice]

Is there a way to adjust the certificate validity window so I don't need to wait 24 hours? For instance set it to 1 hour instead?

[howardjohn]

SECRET_TTL env var on the sidecar should configure it

[bilsch-nice]

How would I go about applying that as the sidecar is injected when the pod is created

[bilsch-nice]

how can I remove the existing certificate to force re-generation? if I kill the pod the replacement pod just assumes the previous certificate. In the past to reset the certs I've uninstalled the istio helm charts and then re-installed which feels like overkill.

I updated the istio-sidecar-injector config map to set the environment variable and confirmed the env var/value is set properly on the new sidecar

[bilsch-nice]

Ok so the issue must have something to do with the laptop going to sleep sorry I thought I accounted for that.

❯ cat istio-proxy.proxy-config.secret.log
RESOURCE NAME     TYPE           STATUS     VALID CERT     SERIAL NUMBER                        NOT AFTER                NOT BEFORE
default           Cert Chain     ACTIVE     true           743cac9a9d57c2b902f010fd579ff85f     2024-04-23T14:30:12Z     2024-04-23T13:28:12Z
ROOTCA            CA             ACTIVE     true           5a16725f5d4a1a1a7b90fb1227db3484     2034-04-07T18:32:21Z     2024-04-09T18:32:21Z
RESOURCE NAME     TYPE           STATUS     VALID CERT     SERIAL NUMBER                        NOT AFTER                NOT BEFORE
default           Cert Chain     ACTIVE     true           743cac9a9d57c2b902f010fd579ff85f     2024-04-23T14:30:12Z     2024-04-23T13:28:12Z
ROOTCA            CA             ACTIVE     true           5a16725f5d4a1a1a7b90fb1227db3484     2034-04-07T18:32:21Z     2024-04-09T18:32:21Z
RESOURCE NAME     TYPE           STATUS     VALID CERT     SERIAL NUMBER                        NOT AFTER                NOT BEFORE
default           Cert Chain     ACTIVE     true           b7363220786cae36ea0b3446ccfbdfe      2024-04-23T15:00:12Z     2024-04-23T13:58:12Z
ROOTCA            CA             ACTIVE     true           5a16725f5d4a1a1a7b90fb1227db3484     2034-04-07T18:32:21Z     2024-04-09T18:32:21Z

And the cert being renewed:

❯ grep -i cert istio-proxy.otel.log
2024-04-23T13:30:11.581277Z     info    Using CA istiod.istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem
2024-04-23T13:30:12.804107Z     info    cache   generated new workload certificate      latency=1.019683276s ttl=59m59.195902731s
2024-04-23T13:30:12.804202Z     info    cache   Root cert has changed, start rotating root cert
2024-04-23T13:30:12.804418Z     info    cache   returned workload certificate from cache        ttl=59m59.195583397s
2024-04-23T14:00:12.764580Z     info    cache   generated new workload certificate      latency=361.238175ms ttl=59m59.235428792s

Is it possible to add a feature request to detect that a cert was expired / not renewed to account for a situation like I'm running where the host running kubernetes/pods may be down temporarily? The only other solution is to delete the pods / let them restart to generate new certs. I know its more of an edge case compared to the normal runtime environment.