Let's Encrypt fails to renew or create certificate, no errors

[visualwritings]

We ran into an issue with Let's Encrypt not renewing our certificate anymore. We're running the latest version of MeshCentral on an Ubunut 20.04LTS machine.

We restarted MC this morning, which resulted in the following output from the leevents command:

4/26/2024 9:55:34 AM - Getting certs from local store (Production)
4/26/2024 9:55:34 AM - Reading certificate files
4/26/2024 9:55:34 AM - Setting LE cert for default domain.
4/26/2024 9:55:34 AM - Setting LE cert for domain dvrsolutions.
4/26/2024 9:55:40 AM - Certificate has -1 day(s) left.
4/26/2024 9:55:40 AM - Asking for new certificate because of expire time.
4/26/2024 9:55:40 AM - Generating private key...
4/26/2024 9:55:41 AM - Setting up ACME client...
4/26/2024 9:55:41 AM - Creating certificate request...
4/26/2024 9:55:41 AM - Requesting certificate from Let's Encrypt...
4/26/2024 9:55:45 AM - Succesful response to challenge.
4/26/2024 9:55:46 AM - Succesful response to challenge.
4/26/2024 9:55:46 AM - Succesful response to challenge.
4/26/2024 9:55:47 AM - Succesful response to challenge.
4/26/2024 9:55:47 AM - Succesful response to challenge.
4/26/2024 9:55:47 AM - Succesful response to challenge.

To check if it might be a permission issue we removed the production.* files from the letsencrypt directory and tried again, with the same result.

We eventually got it sorted by setting skipchallengeverification to true in the config. However, it has been working pretty flawless since 2021, so it is a bit odd this suddenly occurs. No other errors were encounted. I checked the code for LetsEncrypt and found that none of the log-messages that usually should follow "Requesting certificate from Let's Encrypt..." were outputted, so it seems the process silently fails after "Succesful response to challenge".

From what I understand from other related issues here the skipchallengeverification is a self-check used by the letsencrypt module, but I'm entirely sure what the implications are of skipping this check.

In short, for now it is working again, but depending on the cause of the issue I figured you should be made aware of this. Besides that, as mentioned I do not fully understand the implications of skipchallengeverification.

[PathfinderNetworks]

This LetsEncrypt renewal issue is widespread this month due to changes LetsEncrypt has made- especially with the addition of more verification servers located in countries that hadn't been used previously.
By any chance do you use geolocation blocking at your firewall? I do and block most of the world- especially much of Eastern Europe and Asian countries that are hacking hotspots. If so, this is almost certainly the issue you are seeing. I've moved most of my services over to ZeroSSL as a result. And am hoping ZeroSSL support can be added to MeshCentral.

Here is the notice from LetsEncrypt about this:
https://community.letsencrypt.org/t/unexpected-renewal-failures-during-april-2024-please-read-this/216830

[visualwritings]

@PathfinderNetworks I guess it could be related, but the output mentions the certificate was properly retrieved, but it just wasn't saving the cert files. It didn't give any error or notice, the renew / create process just quit without notice.

[PathfinderNetworks]

My apologies, you are correct. That is a completely different issue. The issue I mentioned results in the verification failing the step before a certificate is issued.