Error accepting authorization: acme: authorization error for {MY_DOMAIN}: 500 urn:ietf:params:acme:error:serverInternal: During secondary validation: Remote PerformValidation RPC failed

[marload]

I am trying to generate a letsencrypt TLS cert using cert-manager.
I'm working on AWS, and I need to issue a certificate for DNS registered with Route53. Also I am using external-dns.

For wildcard cert

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: cert-manager-acme-issuer
  # Important: use the same namespace as Cert Manager deployment
  # Otherwise Cert Manager won't be able to find related elements
  namespace: cert-manager
spec:
  acme:
    # Email on which you'll receive notification for our certificates (expiration and such)
    email: MY_EMAIL
    # Name of the secret under which to store the secret key used by acme
    # This secret is managed by ClusterIssuer resource, you don't have to create it yourself
    privateKeySecretRef:
      name: cert-manager-acme-private-key
    # ACME server to use
    # Specify https://acme-v02.api.letsencrypt.org/directory for production
    # Staging server issues staging certificate which won't be trusted by most external parties but can be used for development purposes
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    # Solvers define how to validate you're the owner of the domain for which to issue certificate
    # We use DNS-01 challenge with Route53 by providing related AWS credentials (access key and secret key) for an IAM user with proper rights to manage Route53 records 
    solvers:
    - dns01:
        route53:
          # AWS Access Key ID for our Secret Key
          accessKeyID: MY_ACCESS_KEY
          # AWS region to use 
          region: us-east-1
          # Reference our secret with Secret Key
          secretAccessKeySecretRef:
            key: secret-access-key
            name: cert-manager-aws-secret

Test Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: cert-manager-acme-issuer
    kubernetes.io/ingress.class: nginx
  name: mytest.{MY_DOMAIN}
  namespace: default
spec:
  rules:
  - host: mytest.{MY_DOMAIN}
    http:
      paths:
      - backend:
          service:
            name: chartmuseum
            port:
              number: 8080
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - mytest.{MY_DOMAIN}
    secretName: mytest.{MY_DOMAIN}

When I apply the above yaml, the following error occurs.

cert manager pod

│ E0712 06:15:10.452121       1 controller.go:102] ingress 'default/mytest.{MY_DOMAIN}' in work queue no longer exists                                                                                                                                                                                                                                                                                                                     │
│ E0712 06:15:10.513625       1 controller.go:102] ingress 'default/mytest.{MY_DOMAIN}' in work queue no longer exists                                                                                                                                                                                                                                                                                                                     │
│ I0712 06:15:10.546145       1 util.go:84] cert-manager/controller/certificaterequests-issuer-acme/handleOwnedResource "msg"="owning resource not found in cache" "related_resource_kind"="CertificateRequest" "related_resource_name"="mytest.{MY_DOMAIN}-kb7bk" "related_resource_namespace"="default" "resource_kind"="Order" "resource_name"="mytest.{MY_DOMAIN}-kb7bk-3494100443" "resource_namespace"="default" "resource_version │
│ E0712 06:15:10.546183       1 controller.go:176] cert-manager/orders "msg"="order in work queue no longer exists" "error"="order.acme.cert-manager.io \"mytest.{MY_DOMAIN}-kb7bk-3494100443\" not found"                                                                                                                                                                                                                                 │
│ I0712 06:15:10.567532       1 util.go:84] cert-manager/controller/orders/handleOwnedResource "msg"="owning resource not found in cache" "related_resource_kind"="Order" "related_resource_name"="mytest.{MY_DOMAIN}-kb7bk-3494100443" "related_resource_namespace"="default" "resource_kind"="Challenge" "resource_name"="mytest.{MY_DOMAIN}-kb7bk-3494100443-1383021549" "resource_namespace"="default" "resource_version"="v1"       │
│ E0712 06:15:10.666200       1 controller.go:234] cert-manager/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"mytest.{MY_DOMAIN}-kb7bk-3494100443-1383021549\" not found"                                                                                                                                                                                                          │
│ I0712 06:15:10.666204       1 util.go:84] cert-manager/controller/orders/handleOwnedResource "msg"="owning resource not found in cache" "related_resource_kind"="Order" "related_resource_name"="mytest.{MY_DOMAIN}-kb7bk-3494100443" "related_resource_namespace"="default" "resource_kind"="Challenge" "resource_name"="mytest.{MY_DOMAIN}-kb7bk-3494100443-1383021549" "resource_namespace"="default" "resource_version"="v1"       │
│ I0712 06:15:21.302455       1 conditions.go:201] Setting lastTransitionTime for Certificate "mytest.{MY_DOMAIN}" condition "Ready" to 2022-07-12 06:15:21.302446955 +0000 UTC m=+2853.916736346                                                                                                                                                                                                                                          │
│ I0712 06:15:21.302807       1 trigger_controller.go:200] cert-manager/certificates-trigger "msg"="Certificate must be re-issued" "key"="default/mytest.{MY_DOMAIN}" "message"="Issuing certificate as Secret does not exist" "reason"="DoesNotExist"                                                                                                                                                                                     │
│ I0712 06:15:21.302821       1 conditions.go:201] Setting lastTransitionTime for Certificate "mytest.{MY_DOMAIN}" condition "Issuing" to 2022-07-12 06:15:21.302817103 +0000 UTC m=+2853.917106496                                                                                                                                                                                                                                        │
│ I0712 06:15:21.346760       1 controller.go:161] cert-manager/certificates-readiness "msg"="re-queuing item due to optimistic locking on resource" "key"="default/mytest.{MY_DOMAIN}" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"mytest.{MY_DOMAIN}\": the object has been modified; please apply your changes to the latest version and try again"                                                      │
│ I0712 06:15:21.346991       1 conditions.go:201] Setting lastTransitionTime for Certificate "mytest.{MY_DOMAIN}" condition "Ready" to 2022-07-12 06:15:21.34698535 +0000 UTC m=+2853.961274744                                                                                                                                                                                                                                           │
│ I0712 06:15:21.583677       1 controller.go:161] cert-manager/certificates-key-manager "msg"="re-queuing item due to optimistic locking on resource" "key"="default/mytest.{MY_DOMAIN}" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"mytest.{MY_DOMAIN}\": the object has been modified; please apply your changes to the latest version and try again"                                                    │
│ I0712 06:15:21.613693       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "mytest.{MY_DOMAIN}-gc52c" condition "Approved" to 2022-07-12 06:15:21.613686402 +0000 UTC m=+2854.227975793                                                                                                                                                                                                                          │
│ I0712 06:15:21.644852       1 conditions.go:261] Setting lastTransitionTime for CertificateRequest "mytest.{MY_DOMAIN}-gc52c" condition "Ready" to 2022-07-12 06:15:21.644842782 +0000 UTC m=+2854.259132166                                                                                                                                                                                                                             │
│ E0712 06:15:23.608216       1 controller.go:210] cert-manager/challenges/scheduler "msg"="error scheduling challenge for processing" "error"="Operation cannot be fulfilled on challenges.acme.cert-manager.io \"mytest.{MY_DOMAIN}-gc52c-3494100443-744267503\": the object has been modified; please apply your changes to the latest version and try again" "resource_kind"="Challenge" "resource_name"="mytest.{MY_DOMAIN}-gc52c-3 │
│ E0712 06:18:28.676969       1 sync.go:386] cert-manager/challenges/acceptChallenge "msg"="error waiting for authorization" "error"="acme: authorization error for mytest.{MY_DOMAIN}: 500 urn:ietf:params:acme:error:serverInternal: During secondary validation: Remote PerformValidation RPC failed" "dnsName"="mytest.{MY_DOMAIN}" "resource_kind"="Challenge" "resource_name"="mytest.{MY_DOMAIN}-gc52c-3494100443-744267503" "r │
│ I0712 06:18:28.849491       1 conditions.go:190] Found status change for Certificate "mytest.{MY_DOMAIN}" condition "Issuing": "True" -> "False"; setting lastTransitionTime to 2022-07-12 06:18:28.84948344 +0000 UTC m=+3041.463772824                                                                                                                                                                                                 │
│ I0712 06:18:28.864754       1 trigger_controller.go:179] cert-manager/certificates-trigger "msg"="Backing off from issuance due to previously failed issuance(s). Issuance will next be attempted at 2022-07-12 07:18:28.000000643 +0000 UTC m=+6640.614290026" "key"="default/mytest.{MY_DOMAIN}"                                                                                                                                       │
│ I0712 06:18:28.888288       1 trigger_controller.go:179] cert-manager/certificates-trigger "msg"="Backing off from issuance due to previously failed issuance(s). Issuance will next be attempted at 2022-07-12 07:18:28.000013162 +0000 UTC m=+6640.614302549" "key"="default/mytest.{MY_DOMAIN}"

order

STATE invalid

challenges

STATE invalid
Error accepting authorization: acme: authorization error for mytest.{MY_DOMAIN}: 500 urn:ietf:params:acme:error:serverInternal: During secondary validation: Remote PerformValidation RPC failed

In the end, no certificate is generated. I want a way to solve this.

I always appreciate your contributions. thanks