Race Condition: Cert-Manager Generates Endless Certificate Requests on Openshift

[ykoer]

Describe the bug:
I've encountered some kind of certificate request race condition when I reference an already existing secret that was created by OpenShift's Service-Serving-Certicate. Cert-Manager generates a new certificate request every few seconds until I delete the certificate resource. I could reproduce the issue with the self-signed cluster issuer as well.

Steps to reproduce the bug:

1. Create a selfsigned cluster issuer

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-cluster-issuer
spec:
  selfSigned: {}
EOF

2. Create a Service with the "service.beta.openshift.io/serving-cert-secret-name"
   The annotation is responsible for creating a secret with tls.key and tls.crt, where the subject is CN=test-service.default.svc

kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: test-service
  namespace: default
  annotations:
    service.beta.openshift.io/serving-cert-secret-name: test-service-tls
spec:
  ports:
  - name: https
    port: 8443
  selector:
    app: test-service
EOF

4. Create a Certificate that references the secret that is owned by service

kubectl create -f - <<EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: infinite-certificaterequest-reproducer
  namespace: default
spec:
  issuerRef:
    group: cert-manager.io
    kind: ClusterIssuer
    name: selfsigned-cluster-issuer
  secretName: test-service-tls # <--- reference the secret that is owned by test-service
  commonName: test-service.mydomain.com
  privateKey:
    algorithm: RSA
    size: 3072
    rotationPolicy: Always
    encoding: PKCS1
EOF

5. Watch certificate requests
   You should see a new request every few seconds
   watch kubectl get crs

Environment details::

• Kubernetes Version: v1.27.11+ec42b99
• OpenShift Version: 4.14.20
• cert-manager version: v1.13.1 (cert-manager Operator for Red Hat OpenShift)
• Install method: cert-manager Operator for Red Hat OpenShift

/kind bug

[inteon]

cert-manager Operator for Red Hat OpenShift is not officially distributed by the cert-manager team.
I think you will get better support by reaching out to Red Hat with this issue.
The cert-manager team publishes this OLM module instead: https://cert-manager.io/docs/installation/operator-lifecycle-manager/

[ykoer]

Hello @inteon,
thank you for the prompt response. However, I respectfully disagree with your statement. The issue is entirely reproducible on OpenShift when I manually install cert-manager using the following command:
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.1/cert-manager.yaml

So it doesn't have anything to do with OpenShift Cert-Manager Operator

[inteon]

Hello @inteon,
thank you for the prompt response. However, I respectfully disagree with your statement. The issue is entirely reproducible on OpenShift when I manually install cert-manager using the following command:
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.13.1/cert-manager.yaml

So it doesn't have anything to do with OpenShift Cert-Manager Operator

Thank you for that extra information. Do you have a minimal example that we can use to reproduce the issue (preferably one on a non-openshift cluster).

[hawksight]

@ykoer it appears from the OpenShift docs that Openshift has another controller somewhere that creates the certificate key pair for you and injects it into a secret you specify in the service annotation.

What you are doing on top of this is to then apply a Certificate resource which points to the same Secret resource?

If that is the case this probably isn't a good pattern. Just as it is not good to have two Certificates with different spec fields pointed at the same backend Secret resource.

Please let me know if I am missing something there.

Also to @inteon's point, is the Openshift controller doing the initial creation installer available to install in vanilla k8s clusters. Just running OpenShift environments can be difficult and time consuming to replicate issues.

[ykoer]

@hawksight yes, that's correct. My concern is that it could potentially lead to a disaster if someone mistakenly uses the same secret name in the certificate resource.

@inteon, @hawksight
I have deployed OpenShift Service CA to Kubernetes and have been able to reproduce the issue. You can access all the steps in the Jupyter notebook available in the repository linked below. I hope this clarifies the issue.

https://github.com/ykoer/kubernetes-service-ca

[maelvls]

Hey @ykoer, thanks for the super detailed instructions for reproducing the issue!

The OpenShift team seems to be investigating something close to this in CM-121: "Investigate how to replace Serving certificates generated internally to those generated by cert-manager, which is a Day 2 Operator".

@TrilokGeer Does this issue relate to what you are working on? Thanks!