Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Can't issue an ssl certificate for domains proxied by Cloudflare #258

Open
justemma opened this issue May 27, 2021 · 14 comments
Open

Can't issue an ssl certificate for domains proxied by Cloudflare #258

justemma opened this issue May 27, 2021 · 14 comments

Comments

@justemma
Copy link

justemma commented May 27, 2021 •
edited

Hello!

I was playing around with lua-resty-auto-ssl and everything worked perfectly until I tried issuing a certificate for a domain proxied by Cloudflare (orange cloud).

This is/are the error/s that I'm getting for the domain that is proxied:

2021/05/27 22:49:03 [error] 102203#102203: *20 [lua] ssl_certificate.lua:68: issue_cert(): auto-ssl: failed to obtain lock: timeout, context: ssl_certificate_by_lua*, client: 162.158.62.86, server: 0.0.0.0:443
2021/05/27 22:49:03 [error] 102203#102203: *20 [lua] ssl_certificate.lua:291: auto-ssl: could not get certificate for xyz.example.com - using fallback - failed to get or issue certificate, context: ssl_certificate_by_>
2021/05/27 22:49:03 [error] 102203#102203: *19 open() "/usr/local/openresty/nginx/html/.well-known/acme-challenge/dImjROwMMdY-dfPkAE-edit" failed (2: No such file or directory), client: 162.158.62.86, serv>
2021/05/27 22:49:03 [error] 102203#102203: *5 [lua] lets_encrypt.lua:40: issue_cert(): auto-ssl: dehydrated failed: env HOOK_SECRET=edited HOOK_SERVER_PORT=8999 /usr/l>
startup_hook
Processing xyz.example.com
 + Signing domains...
 + Generating private key...
 + Generating signing request...
 + Requesting new certificate order from CA...
 + Received 1 authorizations URLs from the CA
 + Handling authorization for xyz.example.com
 + 1 pending challenge(s)
 + Deploying challenge tokens...
deploy_challenge
 + Responding to challenge for xyz.example.com authorization...
invalid_challenge
Invalid challenge: DOMAIN=xyz.example.com RESPONSE={
  "type": "http-01",
  "status": "invalid",
  "error": {
    "type": "urn:ietf:params:acme:error:unauthorized",
    "detail": "Invalid response from https://xyz.example.com/.well-known/acme-challenge/dImjROwMMdY-dfPkAE-edit [2606:4700:3035::ac43:8192]: \"\u003chtml\u003e\\r\\n\u003chead\u003e\u003ctitle\u003>    
     "status": 403
  },
  "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/13492436985/xRMMmQ",
  "token": "dImjROwMMdY-dfPkAE-edit",
  "validationRecord": [
    {
      "url": "http://xyz.example.com/.well-known/acme-challenge/dImjROwMMdY-dfPkAE-edit",
      "hostname": "xyz.example.com",
      "port": "80",
      "addressesResolved": [
        "104.21.2.190",
        "172.67.129.146",
        "2606:4700:3035::ac43:8192",
        "2606:4700:3036::6815:2be"
      ],
      "addressUsed": "2606:4700:3035::ac43:8192"
    },
    {
      "url": "https://xyz.example.com/.well-known/acme-challenge/dImjROwMMdY-dfPkAE-edit",
      "hostname": "xyz.example.com",
      "port": "443",
      "addressesResolved": [
        "172.67.129.146",
        "104.21.2.190",
        "2606:4700:3035::ac43:8192",
        "2606:4700:3036::6815:2be"
      ],
      "addressUsed": "2606:4700:3035::ac43:8192"
    }
  ],
  "validated": "2021-05-27T22:48:32Z"
}
 err: nil, context: ssl_certificate_by_lua*, client: 162.158.63.246, server: 0.0.0.0:443
2021/05/27 22:49:03 [error] 102203#102203: *5 [lua] ssl_certificate.lua:97: issue_cert(): auto-ssl: issuing new certificate failed: dehydrated failure, context: ssl_certificate_by_lua*, client: 162.158.63.246, server: 0.0.0.0>
2021/05/27 22:49:03 [error] 102203#102203: *5 [lua] ssl_certificate.lua:53: issue_cert_unlock(): auto-ssl: failed to unlock: lock does not match expected value, context: ssl_certificate_by_lua*, client: 162.158.63.246, server>
2021/05/27 22:49:03 [error] 102203#102203: *5 [lua] ssl_certificate.lua:291: auto-ssl: could not get certificate for xyz.example.com - using fallback - failed to get or issue certificate, context: ssl_certificate_by_l>

Here is my nginx.conf (pretty much the default one but I'm using redis to store the ssl certificates):

user root;

events {
  worker_connections 1024;
}

http {
  access_log /var/log/openresty/access.log;
  error_log /var/log/openresty/error.log;

  lua_shared_dict auto_ssl 10m;
  lua_shared_dict auto_ssl_settings 64k;
  resolver 8.8.8.8 ipv6=off;

  init_by_lua_block {
    auto_ssl = (require "resty.auto-ssl").new()

    auto_ssl:set("renew_check_interval", 86400)
    auto_ssl:set("storage_adapter", "resty.auto-ssl.storage_adapters.redis")
    auto_ssl:set("redis", {
      host = "x.x.x.x",
      auth = "password",
      port = "6379",
      prefix = "ssl"
    })

    auto_ssl:set("allow_domain", function(domain, auto_ssl, ssl_options, renewal)
      return true
    end)
    auto_ssl:init()
  }

  init_worker_by_lua_block {
    auto_ssl:init_worker()
  }

  server {
    listen 443 ssl;

    ssl_certificate_by_lua_block {
      auto_ssl:ssl_certificate()
    }

    ssl_certificate /etc/ssl/resty-auto-ssl-fallback.crt;
    ssl_certificate_key /etc/ssl/resty-auto-ssl-fallback.key;
  }

  server {
    listen 80;

    location /.well-known/acme-challenge/ {
      content_by_lua_block {
        auto_ssl:challenge_server()
      }
    }
  }

  server {
    listen 127.0.0.1:8999;

    client_body_buffer_size 128k;
    client_max_body_size 128k;

    location / {
      content_by_lua_block {
        auto_ssl:hook_server()
      }
    }
  }
}

After trying once to issue the ssl certificate for the proxied domain this is what I have in redis:

1) "ssl:xyz.example.com:challenge:lZsCo9o6iNsNKnTPNGjnUavffxTBI0I-cdwfef"
2) "ssl:xyz.example.com:challenge:XfxXtUQ9y08GIrO8Il3MY5_t94ZOxduYHRg"
3) "ssl:xyz.example.com:challenge:xe9E_szfJ_xcRD70d3uGJGaur5QFH-6co-FLtaGdY"
4) "ssl:xyz.example.com:challenge:BQaUHoAHk1NiauSJNvLmRPSd5eS1FoyK9WEsa1Vp9NA"
5) "ssl:xyz.example.com:challenge:dImjRzwMtdY-ddo2ix-yx8izv_qP_o-cu5KxACveqQo"

And...again, if I use DNS only instead of Proxied the ssl certificate is issued without a problem.

Can somebody help me understand what's happening? Is there something that I can change/implement in order to allow both proxied/dns only domains to get their own ssl certificate?

Thank you! :D
@gjongenelen
Copy link
Contributor

If your site is behind CloudFlare proxy, the best option is to not use Let’s Encrypt at all, but instead to use Cloudflare’s Origin CA: https://blog.cloudflare.com/cloudflare-ca-encryption-origin/

@justemma
Copy link
Author

@gjongenelen I failed to mention that I'm trying to generate certificates for multiple domains and just some of them are on Cloudflare. I apologize for that :( .

Is there a way to generate a certificate for both, proxied and not proxied domains (through Crowdrole or any other dns console/proxy provider) ?

@gjongenelen
Copy link
Contributor

Do you have "Always SSL/HTTPS" option enabled in Cloudflare. Depending on your config, this may create issues as cloudflare will forward acme challenges to your HTTPS handler.

@justemma
Copy link
Author

justemma commented Jun 3, 2021

Oh, turning "Always SSL/HTTPS" off fixed it! Can you give me more info regarding the last part of your answer: "Depending on your config, this may create issues as cloudflare will forward acme challenges to your HTTPS handler" - how can it be configured in a way that would allow me to use "Always SSL/HTTPS"?

Thanky you! :D

@gjongenelen
Copy link
Contributor

You probably have:

location /.well-known/acme-challenge/ {
      content_by_lua_block {
        auto_ssl:challenge_server()
      }
    }

in your http (tcp/80) handler. I think copying this block to your https-handler would fix the issue. Please copy/duplicate and don't remove it from the http-handler, as let's encrypt will still use the http-handler for domains which aren't proxied via cloudflare.

@justemma
Copy link
Author

justemma commented Jun 9, 2021

Hey @gjongenelen , thank you so much for helping me (and hopefully others) but unfortunatelly that one didn't work (copy/paste the acme challenge location block from port 80 to 443) , i still can't issue certificates for domains that have "Always use https" enabled on cloudflare. Any other ideas? Thank you! 😄

@whizzygeeks
Copy link

whizzygeeks commented Aug 9, 2021 •
edited

You probably have:

location /.well-known/acme-challenge/ {
      content_by_lua_block {
        auto_ssl:challenge_server()
      }
    }

in your http (tcp/80) handler. I think copying this block to your https-handler would fix the issue. Please copy/duplicate and don't remove it from the http-handler, as let's encrypt will still use the http-handler for domains which aren't proxied via cloudflare.

This will probably wont work, letsencrypt always connects on http for challenge url. Best way i see it is to introduce dns validation and skip certificate generation for those domains that do not have correct entries. lua-resty-dns module can be used

@gjongenelen
Copy link
Contributor

This will probably wont work, letsencrypt always connects on http for challenge url. Best way i see it is to introduce dns validation and skip certificate generation for those domains that do not have correct entries. lua-resty-dns module can be used

My guess was that the "Always SSL/HTTPS" setting was redirecting the challenges to the https-handler. DNS validation would be a better option indeed.

@arturmkrtchyan
Copy link

Any idea how one could do on-the-fly DNS validations and not try to generate certificates for those proxied calls ? Would it be adding custom code in allow_domain function ?

@benjamindell
Copy link

Sorry to hijack the post, but we're facing a similar issue. We've just moved over from Route53 to CloudFlare and have now turned on DNS proxy. With proxy turned on we get a similar "ERR_SSL_VERSION_OR_CIPHER_MISMATCH" error for custom domains that we pass via our openresty auto SSL EC2 server to have the SSL certificate created.

I've tried turning off the "Always SSL/HTTPS" setting but that didn't seem to make a difference.

We are also doing a DNS lookup in our lua script, so I'm not sure if that makes a difference.

I don't suppose anyone has any suggestions for other things we might want to check?

@whizzygeeks
Copy link

whizzygeeks commented Jan 27, 2023 via email

@benjamindell
Copy link

Amazing, thanks Abhishek. Just wanted to clarify a couple of things if that's OK.

I don't think we actually created a cert and pen on our EC2 SSL server (the one that uses auto SSL to generate SSL certs for our customers custom domains). Or will there be one on the server that im not aware of?

I'd be interested in some technical help if that's OK? Is that something I can pay you for perhaps?

@whizzygeeks
Copy link

Autossl generates a certificate and keeps it on a local server. Location is dependent on configuration. For default file storage you can find in some subidrectory of /etc/resty-auto-ssl location. You can run locate or find command to search the certificate
I need to understand and evaluate the requirements based on the platform. We can start a new communication thread in case you require support, please DM me directly

@benjamindell
Copy link

Thanks - I've just sent you an email to see if you're able to help directly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@benjamindell @arturmkrtchyan @gjongenelen @whizzygeeks @justemma