v1.12.X release Infinite loop with 2 certs with different keystore settings #6992

hawksight · 2024-05-08T07:19:41Z

Describe the bug:

Have two Certificates using the same spec.secretName and have both outputting spec.keystores, but have one with just jks and the other with both jks and pkcs12.

Once the second cert is issued, cert-manager logs will show it continually evaluating the secret. The annotation in the secret will also constantly be changed back and forth between the cert names as show:

kind: Secret
metadata:
  annotations:
    cert-manager.io/alt-names: my-app-4.example.com
    cert-manager.io/certificate-name: **my-app-3**
    cert-manager.io/common-name: my-app-4.example.com
    cert-manager.io/ip-sans: ""
    cert-manager.io/issuer-group: ""
    cert-manager.io/issuer-kind: Issuer
    cert-manager.io/issuer-name: selfsigned
    cert-manager.io/uri-sans: ""
  creationTimestamp: "2024-05-08T06:59:43Z"
  labels:
    controller.cert-manager.io/fao: "true"
  name: my-app-keystores-cert
  namespace: dupe-secret
  resourceVersion: "24111"

kind: Secret
metadata:
  annotations:
    cert-manager.io/alt-names: my-app-4.example.com
    cert-manager.io/certificate-name: **my-app-4**
    cert-manager.io/common-name: my-app-4.example.com
    cert-manager.io/ip-sans: ""
    cert-manager.io/issuer-group: ""
    cert-manager.io/issuer-kind: Issuer
    cert-manager.io/issuer-name: selfsigned
    cert-manager.io/uri-sans: ""
  creationTimestamp: "2024-05-08T06:59:43Z"
  labels:
    controller.cert-manager.io/fao: "true"
  name: my-app-keystores-cert
  namespace: dupe-secret
  resourceVersion: "25400"

Expected behaviour:

An error message on the second object identifying the secret as in use by the first Certificate resource. Such as this:

  Conditions:
    Last Transition Time:  2024-05-08T07:12:14Z
    Message:               Secret was issued for "my-app-3". If this message is not transient, you might have two conflicting Certificates pointing to the same secret.

Steps to reproduce the bug:

New kind cluster with cert-manager

kind create cluster --name example
helm upgrade --install \
  cert-manager jetstack/cert-manager \
  --create-namespace \
  --namespace cert-manager \
  --version v1.12.10 \
  --values install-values.yaml

My values FYI:

installCRDs: true
extraArgs:
  - --feature-gates=ServerSideApply=true

Setup resource:

---
apiVersion: v1
kind: Namespace
metadata:
  labels:
    kubernetes.io/metadata.name: dupe-secret
  name: dupe-secret
spec:
  finalizers:
  - kubernetes
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned
  namespace: dupe-secret
spec:
  selfSigned: {}

Create credential for keystores, This is just "changeme", so is safe to share. Don't use this anywhere.

apiVersion: v1
data:
  password: Y2hhbmdlbWU=
kind: Secret
metadata:
  name: my-app-keystores-cred
  namespace: dupe-secret
type: Opaque

Create first cert

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: my-app-3
  namespace: dupe-secret
spec:
  secretName: my-app-keystores-cert
  issuerRef:
    name: selfsigned
    kind: Issuer
  commonName: "my-app-3.example.com"
  dnsNames:
  - my-app-3.example.com
  keystores:
    jks:
      create: true
      passwordSecretRef:
        name: my-app-keystores-cred
        key: password
    pkcs12:
      create: false
      passwordSecretRef:
        name: my-app-keystores-cred
        key: password

See normal behaviour and cert created.

Now create a second cert that reuses the secretName field but also includes the spec.keystores.pkcs12.create as true

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: my-app-4
  namespace: dupe-secret
spec:
  secretName: my-app-keystores-cert
  issuerRef:
    name: selfsigned
    kind: Issuer
  commonName: "my-app-4.example.com"
  dnsNames:
  - my-app-4.example.com
  keystores:
    jks:
      create: true
      passwordSecretRef:
        name: my-app-keystores-cred
        key: password
    pkcs12:
      create: true
      passwordSecretRef:
        name: my-app-keystores-cred
        key: password

Watch the cert-manager logs as it loops over the keystore being there or not there

k logs cert-manager-5769f79959-5s8fq -n cert-manager -f --since 5m
I0508 06:54:14.894571       1 trigger_controller.go:194] "cert-manager/certificates-trigger: Certificate must be re-issued" key="dupe-secret/my-app-3" reason="DoesNotExist" message="Issuing certificate as Secret does not exist"
I0508 06:54:14.894582       1 conditions.go:203] Setting lastTransitionTime for Certificate "my-app-3" condition "Ready" to 2024-05-08 06:54:14.894571882 +0000 UTC m=+2144.580916060
I0508 06:54:14.894600       1 conditions.go:203] Setting lastTransitionTime for Certificate "my-app-3" condition "Issuing" to 2024-05-08 06:54:14.894596424 +0000 UTC m=+2144.580940601
I0508 06:54:14.909104       1 conditions.go:203] Setting lastTransitionTime for Certificate "my-app-3" condition "Ready" to 2024-05-08 06:54:14.909089799 +0000 UTC m=+2144.595433976
I0508 06:54:15.020261       1 conditions.go:263] Setting lastTransitionTime for CertificateRequest "my-app-3-bzsx4" condition "Approved" to 2024-05-08 06:54:15.020254215 +0000 UTC m=+2144.706598393
I0508 06:54:15.026184       1 conditions.go:263] Setting lastTransitionTime for CertificateRequest "my-app-3-bzsx4" condition "Ready" to 2024-05-08 06:54:15.026178299 +0000 UTC m=+2144.712522435
I0508 06:54:15.034891       1 conditions.go:203] Setting lastTransitionTime for Certificate "my-app-3" condition "Issuing" to 2024-05-08 06:54:15.034885507 +0000 UTC m=+2144.721229685
I0508 06:54:15.035273       1 conditions.go:192] Found status change for Certificate "my-app-3" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2024-05-08 06:54:15.035270465 +0000 UTC m=+2144.721614643
I0508 06:54:15.043826       1 conditions.go:192] Found status change for Certificate "my-app-3" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2024-05-08 06:54:15.043821007 +0000 UTC m=+2144.730165143
I0508 06:54:30.801009       1 conditions.go:203] Setting lastTransitionTime for Certificate "my-app-4" condition "Ready" to 2024-05-08 06:54:30.80099475 +0000 UTC m=+2160.486646359
I0508 06:54:30.801245       1 trigger_controller.go:194] "cert-manager/certificates-trigger: Certificate must be re-issued" key="dupe-secret/my-app-4" reason="SecretMismatch" message="Existing issued Secret is not up to date for spec: [spec.commonName spec.dnsNames]"
I0508 06:54:30.801270       1 conditions.go:203] Setting lastTransitionTime for Certificate "my-app-4" condition "Issuing" to 2024-05-08 06:54:30.801266375 +0000 UTC m=+2160.486917942
I0508 06:54:30.818990       1 trigger_controller.go:194] "cert-manager/certificates-trigger: Certificate must be re-issued" key="dupe-secret/my-app-4" reason="SecretMismatch" message="Existing issued Secret is not up to date for spec: [spec.commonName spec.dnsNames]"
I0508 06:54:30.819015       1 conditions.go:203] Setting lastTransitionTime for Certificate "my-app-4" condition "Issuing" to 2024-05-08 06:54:30.819010667 +0000 UTC m=+2160.504662275
I0508 06:54:30.833856       1 conditions.go:263] Setting lastTransitionTime for CertificateRequest "my-app-4-qm9mr" condition "Approved" to 2024-05-08 06:54:30.833847917 +0000 UTC m=+2160.519499484
I0508 06:54:30.841175       1 conditions.go:263] Setting lastTransitionTime for CertificateRequest "my-app-4-qm9mr" condition "Ready" to 2024-05-08 06:54:30.841167042 +0000 UTC m=+2160.526818609
I0508 06:54:30.850469       1 conditions.go:203] Setting lastTransitionTime for Certificate "my-app-4" condition "Issuing" to 2024-05-08 06:54:30.850460542 +0000 UTC m=+2160.536112109
I0508 06:54:30.851332       1 conditions.go:192] Found status change for Certificate "my-app-4" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2024-05-08 06:54:30.851328125 +0000 UTC m=+2160.536979692
I0508 06:54:30.859849       1 conditions.go:203] Setting lastTransitionTime for Certificate "my-app-4" condition "Issuing" to 2024-05-08 06:54:30.859841625 +0000 UTC m=+2160.545493150
I0508 06:54:30.863079       1 conditions.go:192] Found status change for Certificate "my-app-4" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2024-05-08 06:54:30.863074042 +0000 UTC m=+2160.548725567
I0508 06:58:32.982004       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-4" resource_name="my-app-4" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore key does not contain data"
I0508 06:58:32.988870       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-3" resource_name="my-app-3" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore create disabled"
I0508 06:58:32.992855       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-4" resource_name="my-app-4" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore key does not contain data"
I0508 06:58:32.999203       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-3" resource_name="my-app-3" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore create disabled"
I0508 06:58:33.003948       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-4" resource_name="my-app-4" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore key does not contain data"
I0508 06:58:33.009885       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-3" resource_name="my-app-3" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore create disabled"
I0508 06:58:33.013746       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-4" resource_name="my-app-4" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore key does not contain data"
I0508 06:58:33.019286       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-3" resource_name="my-app-3" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore create disabled"
I0508 06:58:33.022582       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-4" resource_name="my-app-4" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore key does not contain data"
I0508 06:58:33.028113       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-3" resource_name="my-app-3" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore create disabled"
I0508 06:58:33.031310       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-4" resource_name="my-app-4" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore key does not contain data"
I0508 06:58:33.036818       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-3" resource_name="my-app-3" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore create disabled"
I0508 06:58:33.040778       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-4" resource_name="my-app-4" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore key does not contain data"
I0508 06:58:33.046660       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-3" resource_name="my-app-3" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore create disabled"
I0508 06:58:33.049655       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-4" resource_name="my-app-4" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore key does not contain data"
I0508 06:58:33.054485       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-3" resource_name="my-app-3" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore create disabled"
I0508 06:58:33.057713       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-4" resource_name="my-app-4" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore key does not contain data"
I0508 06:58:33.062691       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-3" resource_name="my-app-3" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore create disabled"
I0508 06:58:33.065692       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-4" resource_name="my-app-4" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore key does not contain data"
I0508 06:58:33.071823       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-3" resource_name="my-app-3" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore create disabled"
I0508 06:58:33.074715       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-4" resource_name="my-app-4" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore key does not contain data"
I0508 06:58:33.079850       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-3" resource_name="my-app-3" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore create disabled"
I0508 06:58:33.082736       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-4" resource_name="my-app-4" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore key does not contain data"
I0508 06:58:33.087942       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-3" resource_name="my-app-3" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore create disabled"
I0508 06:58:33.091004       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-4" resource_name="my-app-4" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore key does not contain data"
I0508 06:58:33.095873       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-3" resource_name="my-app-3" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore create disabled"
I0508 06:58:33.098714       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-4" resource_name="my-app-4" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore key does not contain data"
I0508 06:58:33.105310       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-3" resource_name="my-app-3" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore create disabled"
I0508 06:58:33.108172       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-4" resource_name="my-app-4" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore key does not contain data"
I0508 06:58:33.112838       1 secret_manager.go:94] "cert-manager/certificates-issuing: applying Secret data" key="dupe-secret/my-app-3" resource_name="my-app-3" resource_namespace="dupe-secret" resource_kind="Certificate" resource_version="v1" secret="my-app-keystores-cert" message="PKCS12 Keystore create disabled"

Anything else we need to know?:

I will raise another issue, but assuming the spec.keystores is the same, the second cert, "my-app-4" in my case would just take control of the secret.

The main issue here is the permanent log frenzy which will just obliterate the ability to effectively see anything else going on in cert-manager logs. This type of configuration is very easy for tenants to do and doesn't throw up any warning to them. The platform team will then have to deal with the fallout.

Upgrading to v1.14.4 does fix this error immediately. (v1.13.6 is also unaffected.)
We should maybe back port that functionality to v1.12 release as the LTS?
I think v1.12.10 and earlier to be affected by this.

Environment details::

Kubernetes version: 1.29 (kind latest)
Cloud-provider/provisioner: kind
cert-manager version: v1.12.10
Install method: helm

/kind bug

The text was updated successfully, but these errors were encountered:

SgtCoDFish · 2024-05-09T09:45:15Z

I think we documented this as a known issue for https://github.com/cert-manager/cert-manager/releases/tag/v1.12.6 and then it's not documented further - maybe we should add a note about this to all future 1.12.x versions.

If I remember correctly backporting was going to take a lot of effort because of architectural changes between 1.12 and 1.13.

hawksight · 2024-05-09T13:55:02Z

Thanks @SgtCoDFish for the info. I was looking purely on the website for changes around this but did not see it.
I can see it in GitHub as your linked.

But there is nothing about the issue on the releases page for v1.12 here: https://cert-manager.io/docs/releases/release-notes/release-notes-1.12#v1126

This raises another concern about what Long Term Support (LTS) means, but perhaps that's for an in person call / discussion. To take action here, a PR to the cert-manager website reffering to the bug would help others.

Strategies for avoiding spec.secretName collisions otuside of cert-manager?

Perhaps a Kyverno policy might have the option to introspect the environment?
Default the value based on the certificate name so you would end up copying over a cert instead, which might highlight the problem more quickly?

SgtCoDFish · 2024-05-10T14:09:25Z

I've raised cert-manager/website#1477 to rework these docs, because I agree they weren't super clear! I'm not really sure on what might be the best way to handle this outside of cert-manager - if it's possible to do a kyverno thing that sounds great but I don't have a lot of bandwidth atm!

hawksight · 2024-05-13T16:32:27Z

Thanks @SgtCoDFish for correcting / improving the documentation on this.
I spent some time today to figure out the Kyverno solution to this as well.

If you absolutely want to avoid referencing the same secretName from two Certificates here is an example ClusterPolicy that can do that. I have done some simple tests, but do your own testing to validate.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: limits
spec:
  validationFailureAction: Enforce
  rules:
  - name: limit-secretname-refs
    match:
      any:
      - resources:
          kinds:
          - Certificate
          operations:
          - CREATE
          - UPDATE
    context:
    - name: certCount
      apiCall:
        urlPath: "/apis/cert-manager.io/v1/namespaces/{{request.namespace}}/certificates"
        jmesPath: "items[].spec.secretName" 
    validate:
      message: "Only one Certificate is allowed to reference Secret: {{ request.object.spec.secretName }}"
      deny:
        conditions:
          any:
          - key: "{{ request.object.spec.secretName }}"
            operator: AnyIn
            value: "{{ certCount }}"

Example usaged:

> k apply -f ts-cert-1.yaml
certificate.cert-manager.io/ts-cert-1 created

> k apply -f ts-cert-2.yaml
Error from server: error when creating "ts-cert-2.yaml": admission webhook "validate.kyverno.svc-fail" denied the request:

resource Certificate/kyverno-secretname/ts-cert-2 was blocked due to the following policies

limits:
  limit-secretname-refs: 'Only one Certificate is allowed to reference Secret: another-secret-cert'

I will close this out as resolved on the assumptions:

There will be no baked in solution to v1.12.X
You can avoid conflicts in templating / usage
There is another workaround above ^

cert-manager-prow bot added the kind/bug Categorizes issue or PR as related to a bug. label May 8, 2024

SgtCoDFish mentioned this issue May 10, 2024

Rework 1.12 release notes cert-manager/website#1477

Merged

hawksight closed this as completed May 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v1.12.X release Infinite loop with 2 certs with different keystore settings #6992

v1.12.X release Infinite loop with 2 certs with different keystore settings #6992

hawksight commented May 8, 2024

SgtCoDFish commented May 9, 2024

hawksight commented May 9, 2024

SgtCoDFish commented May 10, 2024

hawksight commented May 13, 2024 •

edited

v1.12.X release Infinite loop with 2 certs with different keystore settings #6992

v1.12.X release Infinite loop with 2 certs with different keystore settings #6992

Comments

hawksight commented May 8, 2024

SgtCoDFish commented May 9, 2024

hawksight commented May 9, 2024

SgtCoDFish commented May 10, 2024

hawksight commented May 13, 2024 • edited

hawksight commented May 13, 2024 •

edited