Cert-manager spamming services at .well-known/acme-challenge endpoint #5958

sarasensible · 2023-04-17T15:55:46Z

sarasensible
Apr 17, 2023

I am confused why cert-manager is spamming our services at the .well-known/acme-challenge endpoint.

Every few seconds cert-manager is issuing a request to services on the cluster which have ingress in front of them.

A sample looks like this:

x.x.x.x - - [17/Apr/2023:15:19:24 +0000] "GET /.well-known/acme-challenge/cuRQiNZz5x8mKuLTx-_WLuvE3jHPyDPA3JWPLs8TLSY HTTP/1.1" 404 19 "http://example.com/.well-known/acme-challenge/cuRQiNZz5x8mKuLTx-_WLuvE3jHPyDPA3JWPLs8TLSY" "cert-manager/v1.7.0 (clean)" 320 0.001 [example-service-5000] [] 10.1.x.x:5000 19 0.000 404 402f9469c4b839dc629ea571dea2cee4

As an aside I had v1.7.0 installed but upgraded to 1.11.1, I am not sure why the logs still show v1.7.0.

In the logs I see the following error:

cert-manager/orders "msg"="re-queuing item due to error processing" "error"="ACME client for issuer not initialised/available"

I use Let's Encrypt for my cluster-issuers, which look like the following (with the actual email redacted):

- apiVersion: cert-manager.io/v1
  kind: ClusterIssuer
  metadata:
    name: letsencrypt
  spec:
    acme:
      email: admin@example.com
      preferredChain: ""
      privateKeySecretRef:
        name: letsencrypt
      server: https://acme-v02.api.letsencrypt.org/directory
      solvers:
      - http01:
          ingress:
            class: nginx
  status:
    acme:
      lastRegisteredEmail: admin@example.com
      uri: https://acme-v02.api.letsencrypt.org/acme/acct/434998770
    conditions:
    - lastTransitionTime: "2022-03-03T20:13:07Z"
      message: The ACME account was registered with the ACME server
      observedGeneration: 3
      reason: ACMEAccountRegistered
      status: "True"
      type: Ready
- apiVersion: cert-manager.io/v1
  kind: ClusterIssuer
  metadata:
    name: letsencrypt-pomerium
  spec:
    acme:
      email: admin@example.com
      preferredChain: ""
      privateKeySecretRef:
        name: letsencrypt-pomerium
      server: https://acme-v02.api.letsencrypt.org/directory
      solvers:
      - http01:
          ingress:
            class: pomerium
  status:
    acme:
      lastRegisteredEmail: admin@example.com
      uri: https://acme-v02.api.letsencrypt.org/acme/acct/482747100
    conditions:
    - lastTransitionTime: "2022-04-04T18:11:11Z"
      message: The ACME account was registered with the ACME server
      observedGeneration: 2
      reason: ACMEAccountRegistered
      status: "True"
      type: Ready

I've used kubectl describe to check all cert-manager resources and nothing seems out of the ordinary.

Any help would be greatly appreciated!

Kubernetes version: 1.25
Cert-manager version: 1.11.1

Answered by sarasensible

Apr 18, 2023

@irbekrm I discovered the problem! I have a backup cluster that is issuing challenges against the primary cluster. So these challenges aren't originating in the primary cluster at all. Thanks so much for your help, mystery solved.

View full answer

irbekrm · 2023-04-18T15:18:25Z

irbekrm
Apr 18, 2023
Maintainer

Hi,

thanks for opening the issue.

I am confused why cert-manager is spamming our services at the .well-known/acme-challenge endpoint.

It appears that there are some unsolved Challenge resources in the cluster. Is that the case?

cert-manager/orders "msg"="re-queuing item due to error processing" "error"="ACME client for issuer not initialised/available"

You probably want to find out what Order resource it is, in what status/for what Certificate etc and whether the issuer referenced on the Certificate is available in cluster and is in ready status.

0 replies

sarasensible · 2023-04-18T15:30:40Z

sarasensible
Apr 18, 2023
Author

thanks so much for the response @irbekrm ! I am unable to find any open challenge resources in the cluster:

kubectl get challenges --all-namespaces
No resources found

All Order resources have a valid status and all Certificates are in Ready: True.

The fact that the log shows 1.7.0 makes me wonder whether there could still be an open challenge from before the upgrade that is no longer visible since the CRD has been upgraded. I can't think of any other reason why this would be happening.

0 replies

irbekrm · 2023-04-18T15:35:08Z

irbekrm
Apr 18, 2023
Maintainer

The fact that the log shows 1.7.0

Could you please add full logs (ideally with --v=5 flag to controller)?

0 replies

sarasensible · 2023-04-18T15:48:56Z

sarasensible
Apr 18, 2023
Author

Sure, here it is:

I0417 21:07:45.902243       1 controller.go:138] cert-manager/orders "msg"="starting worker"
I0417 21:07:45.902260       1 controller.go:154] cert-manager/orders "msg"="syncing item" "key"="protect/sensibleweather-io-ingress-zdg7j-3355431889"
I0417 21:07:45.902767       1 controller.go:174] cert-manager/certificaterequests-issuer-vault "msg"="finished processing work item" "key"="opentelemetry-operator-system/opentelemetry-operator-serving-cert-4lz2b"
I0417 21:07:45.902786       1 controller.go:154] cert-manager/certificaterequests-issuer-vault "msg"="syncing item" "key"="tools/argocd-tls-certificate-ntqtk"
I0417 21:07:45.902813       1 sync.go:79] cert-manager/certificaterequests-issuer-vault "msg"="certificate request Ready condition true so skipping processing" "resource_kind"="CertificateRequest" "resource_name"="argocd-tls-certificate-ntqtk" "resource_namespace"="tools" "resource_version"="v1"
I0417 21:07:45.903268       1 sync.go:67] cert-manager/orders "msg"="skipping updating resource as new status == existing status" "resource_kind"="Order" "resource_name"="sensibleweather-io-ingress-zdg7j-3355431889" "resource_namespace"="protect" "resource_version"="v1"
E0417 21:07:45.903290       1 controller.go:167] cert-manager/orders "msg"="re-queuing item due to error processing" "error"="ACME client for issuer not initialised/available" "key"="protect/sensibleweather-io-ingress-zdg7j-3355431889"
I0417 21:07:45.903314       1 controller.go:154] cert-manager/orders "msg"="syncing item" "key"="sandbox/sensibleweather-io-ingress-hf596-1204081961"
I0417 21:07:45.903461       1 controller.go:174] cert-manager/certificaterequests-issuer-vault "msg"="finished processing work item" "key"="tools/argocd-tls-certificate-ntqtk"
I0417 21:07:45.903474       1 controller.go:154] cert-manager/certificaterequests-issuer-vault "msg"="syncing item" "key"="tools/pomerium-ca-rcfbc"
I0417 21:07:45.903499       1 sync.go:79] cert-manager/certificaterequests-issuer-vault "msg"="certificate request Ready condition true so skipping processing" "resource_kind"="CertificateRequest" "resource_name"="pomerium-ca-rcfbc" "resource_namespace"="tools" "resource_version"="v1"
I0417 21:07:45.903745       1 controller.go:174] cert-manager/certificaterequests-issuer-vault "msg"="finished processing work item" "key"="tools/pomerium-ca-rcfbc"
I0417 21:07:45.903753       1 controller.go:154] cert-manager/certificaterequests-issuer-vault "msg"="syncing item" "key"="tools/pomerium-cert-bw9s9"
I0417 21:07:45.903773       1 sync.go:79] cert-manager/certificaterequests-issuer-vault "msg"="certificate request Ready condition true so skipping processing" "resource_kind"="CertificateRequest" "resource_name"="pomerium-cert-bw9s9" "resource_namespace"="tools" "resource_version"="v1"
I0417 21:07:45.904211       1 controller.go:174] cert-manager/certificaterequests-issuer-vault "msg"="finished processing work item" "key"="tools/pomerium-cert-bw9s9"
I0417 21:07:45.904233       1 controller.go:154] cert-manager/certificaterequests-issuer-vault "msg"="syncing item" "key"="monitoring/prometheus-general-tls-rttvg"

The order mentioned - sensibleweather-io-ingress-zdg7j-3355431889 - is valid, so not sure why it might have been referenced there.

0 replies

sarasensible · 2023-04-18T15:55:27Z

sarasensible
Apr 18, 2023
Author

Here is a full describe on that order referenced in the log:

Name:         sensibleweather-io-ingress-zdg7j-3355431889
Namespace:    protect
Labels:       argocd.argoproj.io/instance=protect
Annotations:  cert-manager.io/certificate-name: sensibleweather-io-ingress
              cert-manager.io/certificate-revision: 3
              cert-manager.io/private-key-secret-name: sensibleweather-io-ingress-7h75s
API Version:  acme.cert-manager.io/v1
Kind:         Order
Metadata:
  Creation Timestamp:  2023-01-12T21:30:46Z
  Generation:          1
  Managed Fields:
    API Version:  acme.cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:cert-manager.io/certificate-name:
          f:cert-manager.io/certificate-revision:
          f:cert-manager.io/private-key-secret-name:
        f:labels:
          .:
          f:argocd.argoproj.io/instance:
        f:ownerReferences:
          .:
          k:{"uid":"ca261e81-ed3a-49c1-b932-c94c48b62455"}:
            .:
            f:apiVersion:
            f:blockOwnerDeletion:
            f:controller:
            f:kind:
            f:name:
            f:uid:
      f:spec:
        .:
        f:dnsNames:
        f:issuerRef:
          .:
          f:group:
          f:kind:
          f:name:
        f:request:
      f:status:
        .:
        f:authorizations:
        f:certificate:
        f:finalizeURL:
        f:state:
        f:url:
    Manager:    controller
    Operation:  Update
    Time:       2023-01-12T21:31:11Z
  Owner References:
    API Version:           cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  CertificateRequest
    Name:                  sensibleweather-io-ingress-zdg7j
    UID:                   ca261e81-ed3a-49c1-b932-c94c48b62455
  Resource Version:        214561140
  UID:                     1835488f-79b7-469b-aad5-6dcee5abf764
Spec:
  Dns Names:
    partner.sensibleweather.io
    protect.sensibleweather.io
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   ClusterIssuer
    Name:   letsencrypt
  Request:  <<cert_matter>>
Status:
  Authorizations:
    Challenges:
      Token:        -z7GY4mhJ8LReQpkxFHR1GRyNibkAqg8zajrNC3GpSw
      Type:         http-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/194682274347/pOamMg
      Token:        -z7GY4mhJ8LReQpkxFHR1GRyNibkAqg8zajrNC3GpSw
      Type:         dns-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/194682274347/wXYD3A
      Token:        -z7GY4mhJ8LReQpkxFHR1GRyNibkAqg8zajrNC3GpSw
      Type:         tls-alpn-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/194682274347/3Vxi_g
    Identifier:     partner.sensibleweather.io
    Initial State:  pending
    URL:            https://acme-v02.api.letsencrypt.org/acme/authz-v3/194682274347
    Wildcard:       false
    Challenges:
      Token:        Hx2bXuZ_42JhekN1PxIiLso2pWXa3fVp_nw03_JFMGQ
      Type:         http-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/194682274357/S8tklg
      Token:        Hx2bXuZ_42JhekN1PxIiLso2pWXa3fVp_nw03_JFMGQ
      Type:         dns-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/194682274357/qb70YA
      Token:        Hx2bXuZ_42JhekN1PxIiLso2pWXa3fVp_nw03_JFMGQ
      Type:         tls-alpn-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/194682274357/UPowcQ
    Identifier:     protect.sensibleweather.io
    Initial State:  pending
    URL:            https://acme-v02.api.letsencrypt.org/acme/authz-v3/194682274357
    Wildcard:       false
  Certificate:      <<cert_matter>>
  Finalize URL:     https://acme-v02.api.letsencrypt.org/acme/finalize/434998770/158448161317
  State:            valid
  URL:              https://acme-v02.api.letsencrypt.org/acme/order/434998770/158448161317
Events:             <none>

0 replies

sarasensible · 2023-04-18T16:03:41Z

sarasensible
Apr 18, 2023
Author

@irbekrm I discovered the problem! I have a backup cluster that is issuing challenges against the primary cluster. So these challenges aren't originating in the primary cluster at all. Thanks so much for your help, mystery solved.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cert-manager spamming services at .well-known/acme-challenge endpoint #5958

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 6 comments

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Cert-manager spamming services at .well-known/acme-challenge endpoint #5958

sarasensible Apr 17, 2023

Replies: 6 comments

irbekrm Apr 18, 2023 Maintainer

sarasensible Apr 18, 2023 Author

irbekrm Apr 18, 2023 Maintainer

sarasensible Apr 18, 2023 Author

sarasensible Apr 18, 2023 Author

sarasensible Apr 18, 2023 Author

sarasensible
Apr 17, 2023

irbekrm
Apr 18, 2023
Maintainer

sarasensible
Apr 18, 2023
Author

irbekrm
Apr 18, 2023
Maintainer

sarasensible
Apr 18, 2023
Author

sarasensible
Apr 18, 2023
Author

sarasensible
Apr 18, 2023
Author