Istio sidecar can only request new cert using istio-token

[lukassup]

When using Citadel CA istio sidecars can fetch new certificates using the previous cert while it's valid even when istio-token is expired.

When using istio-csr / cert-manager / Vault new certs can only be fetched while istio-token is valid but old cert appears to be ignored. Or at least it only fetches new certs with a valid istio-token.

Is this behavior expected or not? I've expected for istio sidecar to renew cert with the previous one.

kubernetes version: v1.26.7
istio version: v1.20.2
cert-manager version: v1.12.7
istio-csr version: v0.8.1
vault version: v1.15.2

[RicHincapie]

This is creating issues with Istio's mesh-onboarded VMs. VMs in Istio work different that pods, because the sidecar agent uses K8s provided SA token only to onboard into the mesh, and from there on it relies exclusively on certs:

Virtual machines handle certificates differently than Kubernetes Pods, which use a Kubernetes-provided service account token to authenticate and renew mTLS certificates. Instead, existing mTLS credentials are used to authenticate with the certificate authority and renew certificates. Istio docs source.

While istiod supports both mTLS and JWT to renew certs here:

    // Notice that the order of authenticators matters, since at runtime
    // authenticators are activated sequentially and the first successful attempt
    // is used as the authentication result.
    authenticators := []security.Authenticator{
        &authenticate.ClientCertAuthenticator{},

istio-csr only supports JWT here:

    auther := kubeauth.NewKubeJWTAuthenticator(mesh.NewFixedWatcher(meshcnf), kubeClient, cluster.ID(opts.ClusterID), nil, jwt.PolicyThirdParty)

This creates the following issue with istio-onboarded VMs when trying to renew certs:

error    klog    "msg"="failed to authenticate request" "error"="failed to validate the JWT from cluster \"k8s\": the service account authentication returns an error: [invalid bearer token, service account token has expired, unknown]" "logger"="grpc-server"

I think this needs (1) a warning in docs before more people hit it; (2) support for mTLS for VMs to authenticate with istio-csr.