Lua error when renewing certs

[kpagcha]

(See the actual error at the end of this post)

Everything was working fine. This is my /usr/local/openresty/nginx/conf/nginx.conf file:

http {
    # ...

    # Initial setup tasks.
    init_by_lua_block {
        auto_ssl = (require "resty.auto-ssl").new()

        auto_ssl:set("allow_domain", function(domain)
            -- Do not generate a certificate for our domains
            local regex = [=[^(\S*\.?mysite\.org)$]=]
            if ngx.re.match(domain, regex, "ijo") then
                return true
            end

            -- Custom domains
            if string.find(domain, ".mysite.org") == nil then
                local http = require("resty.http")
                local httpc = http.new()

                httpc:set_timeout(3000)

                local uri = os.getenv("ALLOW_DOMAIN_API")..domain.."/"
                local res, err = httpc:request_uri(uri, {
                      ssl_verify = true,
                      method = "GET"
                 })
                 -- Only allow domain if our endpoint responds with HTTP 200,
                 -- disallow if 404, other response code or error
                 return res and res.status == 200
            end

            return false
        end)

        auto_ssl:init()
    }

    init_worker_by_lua_block {
        auto_ssl:init_worker()
    }

    include ../sites/enabled/*;
}

We have domains of our own as *.mysite.org and third party custom domains. The allow_domain function returns true straight away if the domain is ours and if it is not, it checks with our backend to determine whether it's allowed and a cert should be issued.

Then, our server configuration at /usr/local/openresty/nginx/sites/enabled/mysite.conf/:

upstream app_server {
    server unix:/home/myapp/run/gunicorn.sock fail_timeout=0;
}

server {
    listen 80 default_server;
    listen [::]:80 default_server;

    # Redirect everything to https
    location / {
        return 301 https://$host$request_uri;
    }

    # Endpoint used by auto-ssl for performing domain verification with Let's Encrypt
    location /.well-known/acme-challenge/ {
        content_by_lua_block {
            auto_ssl:challenge_server()
        }
    }
}

server {
    server_name mysite.org www.mysite.org;
    return 301 https://myapp.mysite.org$request_uri;

    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    # auto-ssl block that handles on-the-fly SSL cert generation
    ssl_certificate_by_lua_block {
        auto_ssl:ssl_certificate()
    }

    # dummy certs required for the test to pass
    ssl_certificate /etc/letsencrypt/live/myapp.mysite.org/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/myapp.mysite.org/privkey.pem; # managed by Certbot

    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
}

server {
    server_name myapp.mysite.org proxy.mysite.org;

    listen 443 ssl http2 default_server;
    listen [::]:443 ssl http2 default_server;

    keepalive_timeout 5;
    client_max_body_size 4G;

    access_log /home/myapp/logs/openresty/access.log;
    error_log /home/myapp/logs/openresty/error.log;

    location /static/ {
        alias /home/myapp/myapp/static/;
        expires 365d;
    }

    location /site_media/ {
        alias /home/myapp/myapp/media/;
        expires 365d;
    }

    # checks for static file, if not found proxy to app
    location / {
        try_files $uri @proxy_to_app;
    }

    location @proxy_to_app {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_redirect off;
        proxy_pass http://app_server;
    }

    error_page 502 /502.html;
    location = /502.html {
       root  /home/myapp/myapp/error_page/;
    }

    # auto-ssl block that handles on-the-fly SSL cert generation
    ssl_certificate_by_lua_block {
        auto_ssl:ssl_certificate()
    }

    # dummy certs
    ssl_certificate /etc/letsencrypt/live/myapp.mysite.org/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/myapp.mysite.org/privkey.pem; # managed by Certbot

    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
}

# Internal server running on port 8999 for handling certificate tasks (by auto-ssl)
server {
    listen 127.0.0.1:8999;

    # Increase the body buffer size, to ensure the internal POSTs can always
    # parse the full POST contents into memory
    client_body_buffer_size 128k;
    client_max_body_size 128k;

    location / {
        content_by_lua_block {
            auto_ssl:hook_server()
        }
    }
}

The issue: certs weren't getting renewed for two domains. One of our own (www.mysite.org) and a custom one (www.someone.org). They were expired and just wouldn't get renewed. Restarting the server didn't help.

In the end, I had to delete the cert files /etc/resty-auto-ssl/storage/file/www.mysite.org%3Alatest and /etc/resty-auto-ssl/storage/file/www.someone.org%3Alatest and restart openresty to force new certs to be generated.

Why wasn't resty auto ssl renewing them and what can I do to prevent this from happening again in the future?

Edit:

This is the actual logged error:

2021/09/01 06:16:40 [error] 16675#16675: *3276968 lua entry thread aborted: runtime error: ...sty/luajit/share/lua/5.1/resty/auto-ssl/servers/hook.lua:43: assertion failed!
stack traceback:
coroutine 0:
        [C]: in function 'assert'
        ...sty/luajit/share/lua/5.1/resty/auto-ssl/servers/hook.lua:43: in function 'server'
        .../local/openresty/luajit/share/lua/5.1/resty/auto-ssl.lua:95: in function 'hook_server'
        content_by_lua(scheduler.conf:112):2: in main chunk, client: 127.0.0.1, server: , request: "POST /deploy-cert HTTP/1.1", host: "127.0.0.1:8999"

I am also getting this one, but less commonly:

2021/08/31 00:41:15 [crit] 25738#25738: *2177881 SSL_do_handshake() failed (SSL: error:1414F178:SSL routines:tls1_set_server_sigalgs:no shared signature algorithms error:1417D0E2:SSL routines:tls_process_client_hello:clienthello tlsext) while SSL handshaking, client: 2001:4ca0:108:42::5, server: [::]:443

[knopp]

I'm having same issue (used through https://github.com/Valian/docker-nginx-auto-ssl/ ). This is also not the fist time this happened. REmoving the storage and restarting helps, but this doesn't feel like a reliable solution.

[kpagcha]

Yeah, deleting certs and restarting the server every time this happens isn't really a solution.

I found out the renewal isn't going throught because of this error:

2021/08/30 08:44:11 [error] 8962#8962: *1603936 lua entry thread aborted: runtime error: ...sty/luajit/share/lua/5.1/resty/auto-ssl/servers/hook.lua:43: assertion failed!
stack traceback:
coroutine 0:
        [C]: in function 'assert'
        ...sty/luajit/share/lua/5.1/resty/auto-ssl/servers/hook.lua:43: in function 'server'
        .../local/openresty/luajit/share/lua/5.1/resty/auto-ssl.lua:95: in function 'hook_server'
        content_by_lua(scheduler.conf:112):2: in main chunk, client: 127.0.0.1, server: , request: "POST /deploy-cert HTTP/1.1", host: "127.0.0.1:8999"

It's an error in the very plugin. Has anybody had the same issue?

[eduardo-santos-carvalho]

Hello, I am also experiencing this problem.

All of a sudden, my certificates don't renew anymore and in order for them to be renewed I have to manually remove the expired certificate.

Did you manage to solve it? @knopp @kpagcha

[eduardo-santos-carvalho]

We're getting the error "failed to get ocsp response: failed to validate OCSP response (http://r3.o.lencr.org): OCSP response not successful (6: unauthorized)" for hundreds of domains we have.

At first, this error is being generated because all domains that are already expired have not been renewed and the SSL file is expired inside the default folder (/etc/resty-auto-ssl/storage/file).

Therefore, the OCSP error would only be a final error due to the lack of renewal.

Now, the problem is that the renovation worked perfectly for the last year. It just stopped working.

The only fact we are sure of so far is: only expired SSL files are not being removed from "/etc/resty-auto-ssl/storage/file".

NOTE: If we remove the expired SSL file from the folder and restart openresty, the SSL is generated perfectly (our problem is that there are hundreds a day).

[jbenguira]

We're getting the error "failed to get ocsp response: failed to validate OCSP response (http://r3.o.lencr.org): OCSP response not successful (6: unauthorized)" for hundreds of domains we have.

did you found a fix?

[eduardo-santos-carvalho]

Hi @jbenguira
The solution to this problem was to define a time for the certificates to be renewed, we set it for every 6 hours, one important thing is to check if there is no cron with a problem.
auto_ssl:set("renew_check_interval", 21600)

Within the Dehydrate configuration, we changed the renewal setting which by default is 30 days to 7 days, ie 7 days before the certificate expires it should try to renew. This has been changed to mitigate renewal limit issues.
/usr/local/bin/resty-auto-ssl/dehydrated
RENEW_DAYS="7" # Before 30

It may be necessary to remove the certificate that is showing the OCSP error, for that you need to remove it from this path here, change [YOURDOMAIN] by the domain you want to remove:
/etc/resty-auto-ssl/storage/file/[YOURDOMAIN]%3Alatest

[jbenguira]

Thanks a lot @eduardo-santos-carvalho for the detailed answer :)