Possible bug: "propagation check failed: did not get expected response when querying endpoint"

[janluke]

We had an issue with certificate renewal and I'm trying to figure out what went wrong. The issue happened with cert-manager 1.6.3 (quite old version, I know).

Our original ingress looked like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-api-ingress
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      add_header X-Request-Time $request_time;
    cert-manager.io/cluster-issuer: letsencrypt-production
    acme.cert-manager.io/http01-edit-in-place: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: 2m
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - api-staging.example.com
        - api.example.com
      secretName: example-api-tls
  rules:
    - host: api-staging.example.com
      http:
        paths:
          - pathType: Prefix
            path: "/"
            backend:
              service:
                name: example-api-staging-service
                port:
                  number: 80
    - host: api.example.com
      http:
        paths:
          - pathType: Prefix
            path: "/"
            backend:
              service:
                name: example-api-service
                port:
                  number: 80

Less than one month later, we split the ingresses for production and staging and we added some path-based routing to run an A/B test. We deployed two releases example-api-ingress and example-api-staging-ingress using the following value files and template:

# production.values.yaml
name: "example-api"
host: "api.example.com"
serviceName: "example-api-service"
variantServiceName: "example-api-variant-service"

---

# staging.values.yaml
name: "example-api-staging"
host: "api-staging.example.com"
serviceName: "example-api-staging-service"
variantServiceName: "example-api-variant-staging-service"

---

# Template
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ .Values.name }}-variant-ingress
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      add_header X-Request-Time $request_time;
    cert-manager.io/cluster-issuer: letsencrypt-production
    acme.cert-manager.io/http01-edit-in-place: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: 2m
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - {{ .Values.host }}
      secretName: {{ .Values.name }}-tls
  rules:
    - host: {{ .Values.host }}
      http:
        paths:
          - pathType: ImplementationSpecific
            # /variant     -> /
            # /variant/    -> /
            # /variant/a/b -> /a/b
            path: "/variant(?:/|$)(.*)"
            backend:
              service:
                name: {{ .Values.variantServiceName }}
                port:
                  number: 80
          - pathType: ImplementationSpecific
            # /    -> /
            # /a/b -> /a/b
            path: "/(.*)"
            backend:
              service:
                name: {{ .Values.serviceName }}
                port:
                  number: 80

The old ingress was deleted. When the A/B test was done, we started routing both the / and /variant paths to the same service (example-api-service) and we removed the "variant" services.

This is the cert-manager error we got in cert-manager logs (for both staging and production):

1 sync.go:186] cert-manager/controller/challenges 
"msg"="propagation check failed" 
"error"="did not get expected response when querying endpoint, expected \"{long value here}\" but got: " 
"dnsName"="api-staging.example.com" 
"resource_kind"="Challenge" 
"resource_name"="example-api-tls-6hghz-1295228937-1321418820" 
"resource_namespace"="default" 
"resource_version"="v1" 
"type"="HTTP-01

To solve the issue we removed the variant routing and re-released:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ .Values.name }}-variant-ingress
  annotations:
    nginx.ingress.kubernetes.io/configuration-snippet: |
      add_header X-Request-Time $request_time;
    cert-manager.io/cluster-issuer: letsencrypt-production
    acme.cert-manager.io/http01-edit-in-place: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: 2m
#    nginx.ingress.kubernetes.io/use-regex: "true"
#    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - {{ .Values.host }}
      secretName: {{ .Values.name }}-tls
  rules:
    - host: {{ .Values.host }}
      http:
        paths:
          - pathType: Prefix
            path: "/"
            backend:
              service:
                name: {{ .Values.serviceName }}
                port:
                  number: 80
#          - pathType: ImplementationSpecific
#            # /prefix     ->  /
#            # /prefix/    -> /
#            # /prefix/a/b -> /a/b
#            path: "/variant(?:/|$)(.*)"
#            backend:
#              service:
#                name: {{ .Values.variantServiceName }}
#                port:
#                  number: 80
#          - pathType: ImplementationSpecific
#            # /    -> /
#            # /a/b -> /a/b
#            path: "/(.*)"
#            backend:
#              service:
#                name: {{ .Values.serviceName }}
#                port:
#                  number: 80