Certificate acquisition fails with "Unable to get base domain [...]"

[tisecinfo]

I've read through the documentation extensively, viewed both open and closed issues and even looked at the source code - I cannot figure out what the problem is.

I've censored the domain deliberately.

Setup on an Ubuntu 22.04:

apt-get install certbot python3-pip
pip3 install certbot-plugin-gandi

Content of /root/Gandi.ini with censored API key:

dns_gandi_api_key=aaabbbfffdddeeeff

The key is 100% correct.

Command to aquire the cert:

certbot certonly --authenticator dns-gandi --dns-gandi-credentials /root/Gandi.ini -d subdomain.domain.tld --post-hook "systemctl reload nginx"

Error message on the console:

Requesting a certificate for subdomain.domain.tld
Unable to find or delete the DNS TXT record: Unable to get base domain for "subdomain.domain.tld"

Notable lines from the log file:

2022-11-19 12:03:37,591:INFO:certbot._internal.auth_handler:Performing the following challenges:
2022-11-19 12:03:37,591:INFO:certbot._internal.auth_handler:dns-01 challenge for subdomain.domain.tld
2022-11-19 12:03:37,592:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.gandi.net:443
2022-11-19 12:03:37,844:DEBUG:urllib3.connectionpool:https://api.gandi.net:443 "GET /v5/livedns/domains/subdomain.domain.tld HTTP/1.1" 403 105
2022-11-19 12:03:37,846:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.gandi.net:443
2022-11-19 12:03:38,715:DEBUG:urllib3.connectionpool:https://api.gandi.net:443 "GET /v5/livedns/domains/domain.tld HTTP/1.1" 403 105
2022-11-19 12:03:38,717:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.gandi.net:443
2022-11-19 12:03:39,843:DEBUG:urllib3.connectionpool:https://api.gandi.net:443 "GET /v5/livedns/domains/tld HTTP/1.1" 403 105
2022-11-19 12:03:39,844:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/certbot/_internal/auth_handler.py", line 70, in handle_authorizations
    resps = self.auth.perform(achalls)
  File "/usr/lib/python3/dist-packages/certbot/plugins/dns_common.py", line 67, in perform
    self._perform(domain, validation_domain_name, validation)
  File "/usr/local/lib/python3.10/dist-packages/certbot_plugin_gandi/main.py", line 59, in _perform
    raise errors.PluginError('An error occurred adding the DNS TXT record: {0}'.format(error))
certbot.errors.PluginError: An error occurred adding the DNS TXT record: Unable to get base domain for "subdomain.domain.tld"

2022-11-19 12:03:39,845:DEBUG:certbot._internal.error_handler:Calling registered functions
2022-11-19 12:03:39,845:INFO:certbot._internal.auth_handler:Cleaning up challenges

What's the catch? What else can I do to narrow down the problem?

[HLFH]

Which version is installed?

pip3 show certbot-plugin-gandi

You should have at least version 1.4.0 of certbot-plugin-gandi.
Ubuntu 22.04 provides an older version 1.3.2 that is no longer fully compatible with the updated Gandi API.

The main issue here is you have conflicting package versions.
You should uninstall certbot:

apt remove certbot

And install certbot as well via pip:

pip3 install certbot certbot-plugin-gandi

Shall we close this issue? @obynio It seems as a duplicate of other issues and you have recently specified in your README:

Be careful, installing this plugin with PyPI will also install certbot via PyPI which may conflict with any other certbot already installed on your system.

[obynio]

I'll let it open a bit to wait for an answer. If I don't get one in a few days I'll close it.

[tisecinfo]

Thank you for your answer. "pip3 show certbot-plugin-gandi" yielded version 1.4.1. I've removed certbot via "apt-get purge certbot", reinstalled it via "pip3 install certbot" and tried again, unfortunately certificate acquisition fails exactly the same way.

[HLFH]

@tisecinfo

I don't think /root/Gandi.ini is a good place.

I have put mine in /etc/letsencrypt/renewal/credentials:

cd /etc/letsencrypt/renewal
mkdir credentials
mv /root/Gandi.ini /etc/letsencrypt/renewal/credentials

Are you running as root all the commands?

sudo -s
pip3 install certbot
pip3 install certbot-plugin-gandi
certbot certonly [...]

@obynio

def _get_base_domain(cfg, domain):
    for candidate_base_domain in dns_common.base_domain_name_guesses(domain):
        response = _request(cfg, 'GET', ('domains', candidate_base_domain))
        if response.ok:
            data = _get_json(response)
            fqdn = data.get('fqdn')
            if fqdn:
                return _BaseDomain(fqdn=fqdn)
    return None

We may have to update this code.
Only the HTTP 200 response is handled.
Based on the Gandi LiveDNS API doc, you can get a different HTTP failed response:

• HTTP 403 (Access to the resource is denied. Mainly due to a lack of permissions to access it.) ;
• HTTP 401 (Bad authentication attempt because of a wrong API Key.).

And more importantly, a message parameter that describes the issue.
The error "Unable to get base domain" is too vague and can represent at least two errors, if not more.

[HLFH]

@tisecinfo Could you use this PR? #44

Since it is not merged yet:

In /usr/lib/python3.10/site-packages/certbot_plugin_gandi/, replace gandi_api.py by this:

import requests
import six

from collections import namedtuple
from certbot.plugins import dns_common

_GandiConfig = namedtuple('_GandiConfig', ('api_key', 'sharing_id',))
_BaseDomain = namedtuple('_BaseDomain', ('fqdn', 'error'))

def get_config(api_key, sharing_id):
    return _GandiConfig(api_key=api_key, sharing_id=sharing_id)


def _get_json(response):
    try:
        data = response.json()
    except ValueError:
        return dict()
    return data


def _get_response_message(response, default='<No reason given>'):
    return _get_json(response).get('message', default)


def _headers(cfg):
    return {
        'Content-Type': 'application/json',
        'Authorization': 'Apikey ' + cfg.api_key
    }


def _get_url(*segs):
    return 'https://api.gandi.net/v5/livedns/{}'.format('/'.join(segs))

def _request(cfg, method, segs, **kw):
    headers = _headers(cfg)
    url = _get_url(*segs)
    return requests.request(method, url, headers=headers, params={'sharing_id': cfg.sharing_id}, **kw)


def _get_base_domain(cfg, domain):
    for candidate_base_domain in dns_common.base_domain_name_guesses(domain):
        response = _request(cfg, 'GET', ('domains', candidate_base_domain))
        data = _get_json(response)
        if response.ok:
            fqdn = data.get('fqdn')
            if fqdn:
                return _BaseDomain(fqdn=fqdn,error=None)
        else:
            error = str.lower(data.get('message'))
            return _BaseDomain(fqdn=None,error=error)


def _get_relative_name(base_domain, name):
    suffix = '.' + base_domain.fqdn
    return name[:-len(suffix)] if name.endswith(suffix) else None


def _get_txt_record(cfg, base_domain, relative_name):
    response = _request(cfg, 'GET', ['domains', base_domain.fqdn, 'records', relative_name, 'TXT'])
    if not response.ok:
        return []
    data = _get_json(response)
    vals = data.get('rrset_values')
    if vals:
        return vals
    else:
        return []


def _update_txt_record(cfg, base_domain, relative_name, rrset):
    return _request(cfg, 'PUT', ['domains', base_domain.fqdn, 'records', relative_name, 'TXT'],
                                json={'rrset_values': rrset})


def _update_record(cfg, domain, name, request_runner):

    base_domain = _get_base_domain(cfg, domain)
    error = base_domain.error
    if error is not None:
        return f"{error} Domain: {domain}"
    relative_name = _get_relative_name(base_domain, name)
    if relative_name is None:
        return 'Unable to derive relative name for "{}"'.format(name)

    response = request_runner(base_domain, relative_name)

    return None if response.ok else _get_response_message(response)


def add_txt_record(cfg, domain, name, value):

    def requester(base_domain, relative_name):
        rrset = [value] + _get_txt_record(cfg, base_domain, relative_name)
        return _update_txt_record(cfg, base_domain, relative_name, rrset)

    return _update_record(cfg, domain, name, requester)


def del_txt_record(cfg, domain, name, value):

    def requester(base_domain, relative_name):
        existing = _get_txt_record(cfg, base_domain, relative_name)
        rrset = list(filter(lambda rr: rr.strip('"') != value, existing))
        return _update_txt_record(cfg, base_domain, relative_name, rrset)

    return _update_record(cfg, domain, name, requester)

[tisecinfo]

gandi_api.py was located in /usr/local/lib/python3.10/dist-packages/certbot_plugin_gandi/gandi_api.py for me. I replaced the contents as instructed, ran the certbot command again and got the following console output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for subdomain.domain.tld
Unable to find or delete the DNS TXT record: the resource could not be found. Domain: subdomain.domain.tld
Hook 'post-hook' reported error code 1
Hook 'post-hook' ran with error output:
 nginx.service is not active, cannot reload.
An error occurred adding the DNS TXT record: the resource could not be found. Domain: subdomain.domain.tld
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

/var/log/letsencrypt/letsencrypt.log has the following relevant entries:

2022-12-09 19:56:37,247:INFO:certbot._internal.auth_handler:Performing the following challenges:
2022-12-09 19:56:37,247:INFO:certbot._internal.auth_handler:dns-01 challenge for subdomain.domain.tld
2022-12-09 19:56:37,249:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.gandi.net:443
2022-12-09 19:56:37,454:DEBUG:urllib3.connectionpool:https://api.gandi.net:443 "GET /v5/livedns/domains/subdomain.domain.tld HTTP/1.1" 404 108
2022-12-09 19:56:37,456:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/certbot/_internal/auth_handler.py", line 86, in handle_authorizations
    resps = self.auth.perform(achalls)
  File "/usr/local/lib/python3.10/dist-packages/certbot/plugins/dns_common.py", line 76, in perform
    self._perform(domain, validation_domain_name, validation)
  File "/usr/local/lib/python3.10/dist-packages/certbot_plugin_gandi/main.py", line 59, in _perform
    raise errors.PluginError('An error occurred adding the DNS TXT record: {0}'.format(error))
certbot.errors.PluginError: An error occurred adding the DNS TXT record: the resource could not be found. Domain: subdomain.domain.tld

2022-12-09 19:56:37,456:DEBUG:certbot._internal.error_handler:Calling registered functions
2022-12-09 19:56:37,456:INFO:certbot._internal.auth_handler:Cleaning up challenges
2022-12-09 19:56:37,457:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.gandi.net:443
2022-12-09 19:56:37,825:DEBUG:urllib3.connectionpool:https://api.gandi.net:443 "GET /v5/livedns/domains/subdomain.domain.tld HTTP/1.1" 404 108
2022-12-09 19:56:37,827:WARNING:certbot_plugin_gandi.main:Unable to find or delete the DNS TXT record: the resource could not be found. Domain: subdomain.domain.tld
2022-12-09 19:56:37,827:INFO:certbot.compat.misc:Running post-hook command: systemctl reload nginx

Then I checked the API key again - it is correct and also domain.tld is registered to this Gandi account.

(Ignore the nginx errors - nginx doesn't start up because the certificate doesn't exist since the acquisition fails.)

[HLFH]

@tisecinfo For the so-called subdomain.domain.tld, I see the new error the resource could not be found.
Have you created at least a A Resource Record for subdomain.domain.tld in the DNS that points to the server IP? Try then with 1.4.3 version without my PR as there is some error in my code which I will look later today.

[tisecinfo]

For other reasons I've redeployed the whole machine, still Ubuntu 22.04, installed certbot and this plugin via:

apt-get install python3-pip
pip3 install certbot certbot-plugin-gandi

(The version installed is 1.4.3)
and requested a certificate via

certbot certonly --authenticator dns-gandi --dns-gandi-credentials /root/Gandi.ini -d subdomain.domain.tld --post-hook "systemctl reload nginx"

...and everything worked first try.
I did not create an A-record for subdomain.domain.tld.
I have no idea what changed, as this is the same approach as described in my second post with the exception that a previous version of certbot was installed and then removed via apt-get.

This issue is now resolved for me, but feel free to ask me about details if you want to narrow down potential bugs.

[tisecinfo]

Thank you for creating such a useful plugin and for your assistance troubleshooting this issue.

[obynio]

@HLFH can you remove the draft flag of your PR if you think it's ready for merge ?

[atelierscym]

Hello everybody, and thank you for developping this plugin.
I've got exactly the same issue when trying to create cert :

Unable to find or delete the DNS TXT record: Unable to get base domain for "demo.mondomaine.net"
An error occurred adding the DNS TXT record: Unable to get base domain for "demo.mondomaine.net"

When I check the logs I see some 404 errors reported :

2023-03-27 16:53:25,256:DEBUG:acme.client:Storing nonce: 1DFADurnf-TEqldiKA3Fj2NNkLhknDc65AIJizw-I4jHPRs
2023-03-27 16:53:25,256:INFO:certbot._internal.auth_handler:Performing the following challenges:
2023-03-27 16:53:25,257:INFO:certbot._internal.auth_handler:dns-01 challenge for demo.mondomaine.net
2023-03-27 16:53:25,258:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.gandi.net:443
2023-03-27 16:53:25,375:DEBUG:urllib3.connectionpool:https://api.gandi.net:443 "GET /v5/livedns/domains/demo.mondomaine.net HTTP/1.1" **404** 108
2023-03-27 16:53:25,379:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.gandi.net:443
2023-03-27 16:53:25,631:DEBUG:urllib3.connectionpool:https://api.gandi.net:443 "GET /v5/livedns/domains/mondomaine.net HTTP/1.1" **404** 99
2023-03-27 16:53:25,633:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): api.gandi.net:443
2023-03-27 16:53:25,724:DEBUG:urllib3.connectionpool:https://api.gandi.net:443 "GET /v5/livedns/domains/net HTTP/1.1" **404** 108

I hope this will give you some clues...
I'm impatient someone could help on that issue... I'm available to proceed to deep tests, if required.
Many thanks in advance :-)
Regards,
Cyril