Getting Certificate "1234567890" invalid certificate Issued by: cert-manager.local using HTTP01 Challenge

[marcellodesales]

I'm trying to get an installation in Azure K8s... My previous experiences with Amazon EKS with DNS01 challenge where everything works quickly fine with AWS Route53...

So this is my first time using HTTP01 challenge with the latest versions of the helm charts of nginx-controller and cert-manager. We are NOT using external-dns because our DNS is currently in another server, but we did set A records to point to the Ingress IP address...

Can anybody please give hints on #3579? I've spent more than 10 hrs on the first run and I've been reviewing and I can't find the problem... The generated certificate is weird:

thank you
Marcello

[meyskens]

cert-manager.io/issue-temporary-certificate: true causes the weird certificate while it is struggeling to get one signed using ACME.
For why that is failing i suggest to use https://cert-manager.io/docs/faq/troubleshooting/

[marcellodesales]

@meyskens yeah, I'm not using the temporary-certificate: true on my declarations, it might be injected by cert-manager...

I made a bit of progress: Although I could get a valid certificate, the order object and related did not get removed, the challenge has been running forever with 404 errors... I've gone through the troubleshooting and I couldn't find any difference from the tutorials and I've re-done everything in a brand new cluster with brand new installations of the parts...

Partial Solution

• Setup Nginx ingress controller with RBAC=true

$ helm install nginx-ingress nginx-stable/nginx-ingress -n kube-system --set rbac.create=true

• Installed cert-manager the same way as described in the Issue
• Updated the Ingress resources annotations with the ingress.kubernetes.io/rewrite-target: /

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/issuer: xyz-issuer-westus-ppd-dev
    ingress.kubernetes.io/rewrite-target: /
    kubernetes.io/ingress.class: nginx
    kubernetes.io/ingress.provider: nginx
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/service-upstream: "true"

Nginx Logs still showing the errors

• This might be due to the fact that the order is still there for some reason...

$ kubectl get cert,certificaterequest,order,challenge -n xyz-westus-ppd-dev
NAME                                       READY   SECRET         AGE
certificate.cert-manager.io/my-tls   True    my-tls   2d9h

NAME                                                    READY   AGE
certificaterequest.cert-manager.io/my-tls-cgs52   True    2d9h

NAME                                                       STATE   AGE
order.acme.cert-manager.io/my-tls-cgs52-2593848565   valid   2d9h

• The logs still showing the challenge requests...

ll-known/acme-challenge/A-HGA-Hh9haKSncriygWrlpSMe0JsJ-cdTokY360ECc" "Go-http-client/1.1" "-"
13.87.166.43 - - [18/Jan/2021:15:18:51 +0000] "GET /.well-known/acme-challenge/A-HGA-Hh9haKSncriygWrlpSMe0JsJ-cdTokY360ECc HTTP/1.1" 301 169 "-" "Go-http-client/1.1" "-"
13.87.166.43 - - [18/Jan/2021:15:18:51 +0000] "GET /.well-known/acme-challenge/A-HGA-Hh9haKSncriygWrlpSMe0JsJ-cdTokY360ECc HTTP/1.1" 404 19 "http://xyz-westus-ppd-dev.xyzzzz.com/.well-known/acme-challenge/A-HGA-Hh9haKSncriygWrlpSMe0JsJ-cdTokY360ECc" "Go-http-client/1.1" "-"
^C

Resulting Certificate

• Cert is properly created

$ kubectl describe certificate my-tls -n xyz-westus-ppd-dev

Name:         my-tls
Namespace:    xyz-westus-ppd-dev
Labels:       app.kubernetes.io/name=xyz-receiver
              env=dev
              location=westus
              product=xyz
              type=ppd
Annotations:  acme.cert-manager.io/http01-override-ingress-class: nginx
              acme.cert-manager.io/http01-override-ingress-name: xyz-receiver-westus-ppd-dev
              cert-manager.io/issue-temporary-certificate: true
API Version:  cert-manager.io/v1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2021-01-16T00:23:28Z
  Generation:          1
  Managed Fields:
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
...
    Manager:    controller
    Operation:  Update
    Time:       2021-01-16T00:23:28Z
  Owner References:
    API Version:           extensions/v1beta1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Ingress
    Name:                  xyz-receiver-westus-ppd-dev
    UID:                   bdf12ac0-2746-43b9-b580-9a51c6a42407
  Resource Version:        249898
  Self Link:               /apis/cert-manager.io/v1/namespaces/xyz-westus-ppd-dev/certificates/my-tls
  UID:                     48367d1a-a22b-43b0-8168-2d40a461db83
Spec:
  Dns Names:
    xyz-westus-ppd-dev.xyzzzz.com
  Issuer Ref:
    Group:      cert-manager.io
    Kind:       Issuer
    Name:       xyz-issuer-westus-ppd-dev
  Secret Name:  my-tls
Status:
  Conditions:
    Last Transition Time:  2021-01-16T00:23:28Z
    Message:               Certificate is up to date and has not expired
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Not After:               2021-04-15T05:50:18Z
  Not Before:              2021-01-15T05:50:18Z
  Renewal Time:            2021-03-16T05:50:18Z
Events:                    <none>

• HTTPS endpoint is now taking traffic properly

$ curl -i https://xyz-westus-ppd-dev.xyzzzz.com/admin/health/liveness
HTTP/1.1 200 OK
Server: nginx/1.19.3
Date: Mon, 18 Jan 2021 15:12:34 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 15
Connection: keep-alive
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept, Authorization, Access-Control-Request-Method
Access-Control-Allow-Origin: *
X-proct-Module: xyz-receiver
X-proct-Version: 563030f
X-proct-Tid: 1c096b76-5188-43c7-8c73-f5dc1e7fccd6

{"status":"UP"}%

• Cert

$ echo | openssl s_client -showcerts -servername xyz-westus-ppd-dev.xyzzzz.com -connect xyz-westus-ppd-dev.xyzzzz.com:443 2>/dev/null | openssl x509 -inform pem -noout -text

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:c1:ad:60:7c:57:4d:5d:fa:13:ae:76:de:96:e8:dc:f9:51
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Let's Encrypt, CN = R3
        Validity
            Not Before: Jan 16 04:38:17 2021 GMT
            Not After : Apr 16 04:38:17 2021 GMT
        Subject: CN = xyz-westus-ppd-dev.xyzzzz.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:b9:6b:a9:fc:4c:48:2c:94:f5:18:49:df:fe:74:
                    55:af:26:88:08:df:c9:4e:c9:d4:68:fb:82:93:b2:
                    9e:09:6a:d5:3e:28:e6:e4:d8:a2:85:68:ee:01:ee:
                    f8:0a:45:f6:bb:24:c4:32:69:eb:a2:7d:28:3f:ee:
                    e3:7d:61:33:5e:37:02:ad:69:90:06:8a:99:1b:69:
                    7a:b0:0b:b8:1a:ca:3f:cc:3f:89:b8:44:86:80:99:
                    b4:2f:df:e2:d2:c9:b0:c5:4c:18:79:d0:c7:ae:73:
                    db:d6:17:b6:08:58:3c:c0:84:ef:33:b1:ea:42:69:
                    c3:b1:ec:f8:a6:c5:ee:3f:d5:bc:74:c8:f6:5b:20:
                    de:e2:39:51:58:74:db:0e:41:05:b1:f0:82:f4:f6:
                    4b:f4:60:54:fa:6b:33:ad:15:f1:4b:7a:a6:01:77:
                    89:bb:82:0a:11:33:9b:d6:00:61:a4:e2:20:4c:51:
                    e5:09:9b:2c:3a:5b:59:f7:72:09:8c:00:81:c8:cd:
                    b2:7b:47:e3:fe:22:13:43:29:35:21:a2:b7:32:d9:
                    c7:c5:84:a4:3d:ca:a2:57:b4:b3:44:c4:10:56:a5:
                    aa:9c:07:21:9a:f8:0e:2d:76:61:d3:b5:1b:da:f1:
                    63:a5:5b:51:ec:e2:3c:9d:c8:02:a3:c5:af:96:f7:
                    36:63
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                24:E4:18:5C:73:A9:E6:E3:24:A3:C9:D3:11:02:C5:8B:DB:1C:E8:BA
            X509v3 Authority Key Identifier:
                keyid:14:2E:B3:17:B7:58:56:CB:AE:50:09:40:E6:1F:AF:9D:8B:14:C2:C6

            Authority Information Access:
                OCSP - URI:http://r3.o.lencr.org
                CA Issuers - URI:http://r3.i.lencr.org/

            X509v3 Subject Alternative Name:
                DNS:xyz-westus-ppd-dev.xyzzzz.com
            X509v3 Certificate Policies:
                Policy: 2.23.140.1.2.1
                Policy: 1.3.6.1.4.1.44947.1.1.1
                  CPS: http://cps.letsencrypt.org

            CT Precertificate SCTs:
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 6F:53:76:AC:31:F0:31:19:D8:99:00:A4:51:15:FF:77:
                                15:1C:11:D9:02:C1:00:29:06:8D:B2:08:9A:37:D9:13
                    Timestamp : Jan 16 05:38:17.413 2021 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:21:00:96:72:2F:47:EE:4A:C5:CF:2D:12:47:
                                22:F5:30:14:3C:67:2A:44:D4:00:91:15:8E:7F:96:30:
                                EE:21:94:E0:99:02:20:1F:59:13:79:EB:67:E4:E0:93:
                                E4:8E:C9:AA:AC:2A:EC:7D:BE:5F:F9:49:B5:64:70:1A:
                                17:97:8B:5C:47:10:A8
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 7D:3E:F2:F8:8F:FF:88:55:68:24:C2:C0:CA:9E:52:89:
                                79:2B:C5:0E:78:09:7F:2E:6A:97:68:99:7E:22:F0:D7
                    Timestamp : Jan 16 05:38:17.435 2021 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:44:02:20:1B:00:54:82:C5:CA:57:14:AB:E0:6F:A9:
                                42:96:F2:0D:0A:03:CF:A5:92:3E:7D:1B:49:5C:81:3B:
                                22:DB:C7:64:02:20:6D:E2:37:C3:11:D1:9A:9C:27:13:
                                8B:A3:51:1A:61:7F:D3:6A:B0:1D:37:FC:40:A8:91:AF:
                                EE:FA:63:9D:D3:88
    Signature Algorithm: sha256WithRSAEncryption
         38:bb:25:f7:c2:43:47:8b:c7:0f:2a:1b:59:20:6d:96:14:fe:
         f1:da:b3:13:21:e4:25:a5:f9:4d:3a:1f:2d:a8:82:38:30:bf:
         02:77:47:e5:6b:4a:20:71:b8:6f:be:99:7f:f8:8d:2d:6e:4a:
         95:73:67:36:38:f7:05:cf:00:49:fd:87:18:c1:6b:a5:98:71:
         68:25:03:fb:4c:55:fe:3c:e4:1f:0d:94:1d:1b:ed:c6:0e:7d:
         20:8c:1d:c6:6e:a5:27:76:16:c1:c8:23:dc:50:86:5c:a2:0b:
         80:9c:43:26:ea:d5:9e:82:58:b8:b2:45:63:a6:3e:4c:d1:bf:
         64:8a:e3:bd:e1:a5:1e:27:43:3b:c1:82:fd:a6:6e:82:62:80:
         98:a5:25:c2:f8:f3:cf:2d:fa:4a:89:85:e3:af:4f:00:ca:59:
         6f:cb:01:27:78:1f:9d:bb:a7:5b:93:db:45:6e:e8:17:61:0a:
         5e:71:e9:1c:33:25:86:75:d9:58:2d:4c:1b:e9:f0:52:e0:bb:
         74:24:77:da:1d:0a:cb:74:e8:b4:80:05:16:91:d6:96:c8:ba:
         bb:97:f9:ee:c0:dc:87:91:e1:53:54:04:ee:ce:83:44:e7:21:
         40:ba:5f:2c:32:75:44:de:49:a5:18:65:87:c4:24:69:d2:29:
         43:90:18:62

Questions:

• Any clues about why there's still a challenge being requested?
• Should we just delete the cert-manager's objects for the cert request?