jrcs/letsencrypt-nginx-proxy-companion - HTTPS is throwing 500 Internal Server Error #959

Swapratim · 2022-06-09T12:23:13Z

Hello,

I'm stuck with this prod issue for couple of days and it is hurting a customer now.
So if anyone would be kind enough to help - I'd really appreciate here :-)

I'm using jrcs/letsencrypt-nginx-proxy-companion for generating SSL certs for a number of companion apps running on sidecar docker-compose. There are more than 100 dockers and each one is running with a subdomain.

I pulled the latest docker tag of jrcs/letsencrypt-nginx-proxy-companion couple of days ago and issue started from there.

Issue:

More than half of the apps are running fine with HTTPS. Valid Letsencrypt cert is showing there.
Few apps (docker-compose) are not running with HTTPS - showing "500 Internal Server Error | nginx/1.19.3" error
But they are running fine with HTTP
Lots of subdomains are getting SSL renewed when /app/force_renew is run. But some of them are not getting renewed

Create new order error. Le_OrderFinalize not found. {
  "type": "urn:ietf:params:acme:error:rateLimited",
  "detail": "Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: friis1.marvinxr.com: see https://letsencrypt.org/docs/rate-limits/",
  "status": 429
}

While some of them are auto-renewed (all of them have same configuration)

Creating/renewal spiderman.marvinxr.com certificates... (spiderman.marvinxr.com)
[Thu Jun  9 13:53:29 UTC 2022] Domains not changed.
[Thu Jun  9 13:53:29 UTC 2022] Skip, Next renewal time is: Mon Aug  8 09:00:49 UTC 2022

I have tried restarting/force_renew certificates quite some time (I know it's not the best practice but the client has a big launch and cannot wait). The issue is not getting solved at all.

jrcs/letsencrypt-nginx-proxy-companion docker-compose:

version: '2'

services:

  nginx-proxy:
    image: jwilder/nginx-proxy
    container_name: nginx-proxy
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - conf:/etc/nginx/conf.d
      - vhost:/etc/nginx/vhost.d
      - html:/usr/share/nginx/html
      - dhparam:/etc/nginx/dhparam
      - certs:/etc/nginx/certs:ro
      - /var/run/docker.sock:/tmp/docker.sock:ro
    restart: always

  letsencrypt:
    image: jrcs/letsencrypt-nginx-proxy-companion
    container_name: nginx-proxy-le
    depends_on:
      - nginx-proxy
    volumes_from:
      - nginx-proxy
    volumes:
      - certs:/etc/nginx/certs
      - acme:/etc/acme.sh
      - /var/run/docker.sock:/var/run/docker.sock:ro
    restart: always

volumes:
  conf:
  vhost:
  html:
  dhparam:
  certs:
  acme:

networks:
  default:
    external:
      name: letsencrypt

sidecar application docker-compose:

version: '3.4'

networks:
  application_network:
services:
    ar_app:
      restart: "always"
      container_name: friis1
      build:
        context: app
      environment:
         VIRTUAL_HOST: friis1.marvinxr.com
         PORT: 6000
         LETSENCRYPT_EMAIL: support@marvinai.live
         LETSENCRYPT_HOST: friis1.marvinxr.com
      expose:
        - "6000"


networks:
  default:
    external:
      name: letsencrypt

Dockerfile for application docker-compose (from above)

FROM python:3.11.0b3-alpine3.16

WORKDIR /app

COPY requirements.txt ./

RUN pip install -r requirements.txt

COPY . /app

CMD ["python","-u","server.py"]

I can see the full list of certificate chain and acme keys stored corrctly.
friis1.marvinxr.com (directory)

friis1.marvinxr.com.conf
friis1.marvinxr.com.csr
friis1.marvinxr.com.csr.conf
friis1.marvinxr.com.key

friis1.marvinxr.com.chain.pem
friis1.marvinxr.com.crt
friis1.marvinxr.com.dhparam.pem
friis1.marvinxr.com.key

The certificate was working perfectly for over a month. But then they got regenerated on June 7th.
The problem starts from there.

Doing curl gives me this result:

curl https://friis1.marvinxr.com
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Mozilla says Error code: MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT
https://friis1.marvinxr.com/

The certificate is not trusted because it is self-signed.

HTTP Strict Transport Security: false
HTTP Public Key Pinning: false

Certificate chain:

Then I check it from https://www.ssllabs.com --> it says the cert is invalid

My question is: how to recreate them and make it work asap?

P.S. - It's an AWS server and the inbound security rules are as per the documentation - allowed
I have gone through almost all issues here and I have checked the pre-requisites multiple times already (inluding DNS, wildcard etc.)

Please help. I'm quite clueless now. Maybe a small thing that I'm missing here.

Thanks in advance for your help.

The text was updated successfully, but these errors were encountered:

uacaman · 2022-06-10T03:25:15Z

I have the same problem, i looks like the certificates are generate every time the service is started, even if you use named volumes

Swapratim · 2022-06-10T06:36:04Z

Yes correct. And that makes me little concerned because for production usage there is a rate limit of 5 cert renewal per week. If I have 100 certs failing - how will it ensure that after 7 days all of them will be renewed properly. There's no way I can prioritize anything as such.

Between have you found any solution @uacaman?

Swapratim · 2022-06-10T06:40:59Z

@JrCs and @buchdag - any idea or suggestion please? The client is really not happy since they ordered a custom stencil with QR code with HTTPS link to spray paint it across the city and now it is of no use to them due to the problem.

uacaman · 2022-06-10T11:59:23Z

post your config, maybe i can help. in my case was a typo on volume mappings

buchdag · 2022-06-10T14:03:42Z

@Swapratim could you try to switch to Zero SSL instead of Let's Encrypt ? Zero SSL does not have rate limits. Please check the docs for instructions.

Swapratim · 2022-06-10T14:16:28Z

Thanks @buchdag. I'd rather prefer to Letsencrypt than Zero SSL since their review is not very good (trustpilot).
I'm wondering two questions:

If the cert renew limit is 5/week and all certs gets renewed at new letsencrypt-nginx-proxy-companion start - then how rest of the 100 certs are working fine?
The cert for friis1.marvinxr.com was renewed on June 7th. What made it a bad/invalid certificate while others remain functional?

I compared the issued certs with other containers' certs - and all the formats are same actually.

Swapratim · 2022-06-10T14:17:09Z

I have the same problem, i looks like the certificates are generate every time the service is started, even if you use named volumes

@uacaman I have posted all configs above.

buchdag · 2022-06-10T14:25:07Z

As a temporary / emergency fix you can still obtain a certificate from an outside source and mount it inside the certificate volume for use with nginx-proxy, following the later's doc.

buchdag · 2022-06-10T14:27:17Z

On a side note I'd advise you to never ever use the latest tag of any container on a production environment.

Swapratim · 2022-06-10T14:29:04Z

On a side note I'd advise you to never ever use the latest tag of any container on a production environment.

What is the most stable container version you suggest for jrcs/letsencrypt-nginx-proxy-companion and nginx-proxy?

buchdag · 2022-06-10T14:31:01Z

I'd rather prefer to Letsencrypt than Zero SSL

Since you've blown through rate limiting you're pretty much out of that option if you want to fix this quickly. You can set up Zero SSL on a per container basis and keep Let's Encrypt for the others.

uacaman · 2022-06-10T16:02:06Z

are you using compose version 2? i am not familiar with that syntax, but you have to make sure that the volumes don't get created every time.

In my case, i map the volume to the host like this:

volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /opt/nginx/certs:/etc/nginx/certs
- /opt/nginx/vhost:/etc/nginx/vhost.d
- /opt/nginx/html:/usr/share/nginx/html
- /opt/nginx/acme:/etc/acme.sh

Swapratim · 2022-06-10T16:57:41Z

I had the diaster with version 3. So decided to move it back to version 2 since I was running ver. 2 over an year now.
Volumes are mapped and are only getting created once.

uacaman · 2022-06-10T17:24:50Z

if the volumes are mapped, check the volumes, the generated certificates should be in there.

Swapratim · 2022-06-10T17:56:44Z

If you see above, I have already described that the certs are there. But I provided the cert details too. But somehow one is not working.

uacaman · 2022-06-10T18:05:26Z

and if you connect to the docket and check the file system, the certs are there? Since you hit the rate limit, the only option is to find the already generated certs and use or wait the 7 days

Swapratim · 2022-06-11T06:17:31Z

Thanks @buchdag. I'd rather prefer to Letsencrypt than Zero SSL since their review is not very good (trustpilot). I'm wondering two questions:

If the cert renew limit is 5/week and all certs gets renewed at new letsencrypt-nginx-proxy-companion start - then how rest of the 100 certs are working fine?

The cert for friis1.marvinxr.com was renewed on June 7th. What made it a bad/invalid certificate while others remain functional?

I compared the issued certs with other containers' certs - and all the formats are same actually.

@uacaman That is my problem. I have the certs, they seem legit and got renewed 4 days back by letsencrypt. But it is showing as invalid against the docker-app - when opened from the browser. That is the reason I wanted to ask initially. Maybe I was not clear enough before.

hadpro24 · 2022-07-10T20:51:22Z

Thanks @buchdag. I'd rather prefer to Letsencrypt than Zero SSL since their review is not very good (trustpilot). I'm wondering two questions:

If the cert renew limit is 5/week and all certs gets renewed at new letsencrypt-nginx-proxy-companion start - then how rest of the 100 certs are working fine?

The cert for friis1.marvinxr.com was renewed on June 7th. What made it a bad/invalid certificate while others remain functional?

I compared the issued certs with other containers' certs - and all the formats are same actually.

@uacaman That is my problem. I have the certs, they seem legit and got renewed 4 days back by letsencrypt. But it is showing as invalid against the docker-app - when opened from the browser. That is the reason I wanted to ask initially. Maybe I was not clear enough before.

I have the same probleme. Do you find solution ?

wekaz · 2022-09-04T11:58:14Z

does anyone have solution yet? i thought it's just me

edruder · 2022-12-24T04:41:26Z

I'm having a very similar situation to what's described here. acme-companion is getting a 503 error when GET /.well-known/acme-challenge/<random token>. (A static site also behind nginx-proxy is returning 404s.)

I'm using docker-compose, "three containers example", essentially copied from the wiki.
I'm using zerossl.com because I got rate-limited by letsencrypt.
nginx log for the acme-challenge against my app:
"GET /.well-known/acme-challenge/UjkSifSqQtA67Decxn-jA7xn2O6dlC0OSl39I8JqtyY HTTP/1.1" 503 190 "-" "acme.zerossl.com/v2/DV90" "-"
I've disable IPv6 for my domain.
docker run -d -p 80:80 nginx:alpine on my remote box returns the expected Nginx page.
https://unboundtest.com/ gets validation success

My docker-compose.yml:

version: '2'

services:
  nginx-proxy:
    image: nginx:alpine
    container_name: nginx-proxy
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - conf:/etc/nginx/conf.d
      - vhost:/etc/nginx/vhost.d
      - html:/usr/share/nginx/html
      - certs:/etc/nginx/certs:ro
    network_mode: bridge

  nginx-proxy-gen:
    image: nginxproxy/docker-gen:latest
    container_name: nginx-proxy-gen
    command: -notify-sighup nginx-proxy -watch /etc/docker-gen/templates/nginx.tmpl /etc/nginx/conf.d/default.conf
    volumes_from:
      - nginx-proxy
    volumes:
      - ./nginx.tmpl:/etc/docker-gen/templates/nginx.tmpl:ro
      - /var/run/docker.sock:/tmp/docker.sock:ro
    labels:
      - "com.github.jrcs.letsencrypt_nginx_proxy_companion.docker_gen"
    network_mode: bridge

  nginx-proxy-acme:
    image: nginxproxy/acme-companion:latest
    container_name: nginx-proxy-acme
    volumes_from:
      - nginx-proxy
    volumes:
      - certs:/etc/nginx/certs:rw
      - acme:/etc/acme.sh
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      - DEFAULT_EMAIL=foo@bar.com
      - ACME_CA_URI=https://acme.zerossl.com/v2/DV90
      - ZEROSSL_API_KEY=<something something>
    network_mode: bridge

volumes:
  acme:
  certs:
  conf:
  html:
  vhost:

Any suggestions?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

jrcs/letsencrypt-nginx-proxy-companion - HTTPS is throwing 500 Internal Server Error #959

jrcs/letsencrypt-nginx-proxy-companion - HTTPS is throwing 500 Internal Server Error #959

Swapratim commented Jun 9, 2022 •

edited

uacaman commented Jun 10, 2022

Swapratim commented Jun 10, 2022 •

edited

Swapratim commented Jun 10, 2022

uacaman commented Jun 10, 2022

buchdag commented Jun 10, 2022

Swapratim commented Jun 10, 2022

Swapratim commented Jun 10, 2022

buchdag commented Jun 10, 2022

buchdag commented Jun 10, 2022

Swapratim commented Jun 10, 2022

buchdag commented Jun 10, 2022

uacaman commented Jun 10, 2022

Swapratim commented Jun 10, 2022 •

edited

uacaman commented Jun 10, 2022

Swapratim commented Jun 10, 2022

uacaman commented Jun 10, 2022

Swapratim commented Jun 11, 2022

hadpro24 commented Jul 10, 2022

wekaz commented Sep 4, 2022

edruder commented Dec 24, 2022 •

edited

jrcs/letsencrypt-nginx-proxy-companion - HTTPS is throwing 500 Internal Server Error #959

jrcs/letsencrypt-nginx-proxy-companion - HTTPS is throwing 500 Internal Server Error #959

Comments

Swapratim commented Jun 9, 2022 • edited

uacaman commented Jun 10, 2022

Swapratim commented Jun 10, 2022 • edited

Swapratim commented Jun 10, 2022

uacaman commented Jun 10, 2022

buchdag commented Jun 10, 2022

Swapratim commented Jun 10, 2022

Swapratim commented Jun 10, 2022

buchdag commented Jun 10, 2022

buchdag commented Jun 10, 2022

Swapratim commented Jun 10, 2022

buchdag commented Jun 10, 2022

uacaman commented Jun 10, 2022

Swapratim commented Jun 10, 2022 • edited

uacaman commented Jun 10, 2022

Swapratim commented Jun 10, 2022

uacaman commented Jun 10, 2022

Swapratim commented Jun 11, 2022

hadpro24 commented Jul 10, 2022

wekaz commented Sep 4, 2022

edruder commented Dec 24, 2022 • edited

Swapratim commented Jun 9, 2022 •

edited

Swapratim commented Jun 10, 2022 •

edited

Swapratim commented Jun 10, 2022 •

edited

edruder commented Dec 24, 2022 •

edited