Error in certbot-dns-ovh using lexicon

[mguyard]

Hello,

I use certbot-dns-ovh who use lexicon for dns.
But lexicon return an error for OVH provider. This worked before so i think it's can be a change in OVH api or in lexicon code.

Certbot require lexion >=3.14.1 : https://github.com/certbot/certbot/blob/d9d825ac50414ba837fdd319beaa5a9f72139457/certbot-dns-ovh/setup.py#L10C24-L10C24

As you can see in logs, lexicon return error

2023-10-12 17:11:48,231:DEBUG:certbot._internal.main:certbot version: 2.7.1
2023-10-12 17:11:48,231:DEBUG:certbot._internal.main:Location of certbot entry point: /usr/local/bin/certbot
2023-10-12 17:11:48,231:DEBUG:certbot._internal.main:Arguments: ['--dns-ovh', '--dns-ovh-credentials', '/etc/letsencrypt/.ovhapi', '--non-interactive', '--agree-tos', '--email', 'marc@mydomain.com', '-d', '*.mydomain.com', '--logs-dir', '/etc/letsencrypt/logs', '--config-dir', '/etc/letsencrypt/config', '--work-dir', '/etc/letsencrypt/work', '-v']
2023-10-12 17:11:48,231:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#dns-ovh,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2023-10-12 17:11:48,318:DEBUG:certbot._internal.log:Root logging level set at 20
2023-10-12 17:11:48,321:DEBUG:certbot._internal.plugins.selection:Requested authenticator dns-ovh and installer None
2023-10-12 17:11:48,322:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * dns-ovh
Description: Obtain certificates using a DNS TXT record (if you are using OVH for DNS).
Interfaces: Authenticator, Plugin
Entry point: EntryPoint(name='dns-ovh', value='certbot_dns_ovh._internal.dns_ovh:Authenticator', group='certbot.plugins')
Initialized: <certbot_dns_ovh._internal.dns_ovh.Authenticator object at 0xffff8fefa410>
Prep: True
2023-10-12 17:11:48,322:DEBUG:certbot._internal.plugins.selection:Selected authenticator <certbot_dns_ovh._internal.dns_ovh.Authenticator object at 0xffff8fefa410> and installer None
2023-10-12 17:11:48,322:INFO:certbot._internal.plugins.selection:Plugins selected: Authenticator dns-ovh, Installer None
2023-10-12 17:11:48,367:DEBUG:certbot._internal.main:Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_agreed=None, only_return_existing=None, external_account_binding=None), uri='https://acme-v02.api.letsencrypt.org/acme/acct/728015997', new_authzr_uri=None, terms_of_service=None), 17a77fe8ee348e150aac144f5edce683, Meta(creation_dt=datetime.datetime(2022, 9, 12, 8, 25, tzinfo=<UTC>), creation_host='3bbceec2dec9', register_to_eff=None))>
2023-10-12 17:11:48,368:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2023-10-12 17:11:48,369:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org:443
2023-10-12 17:11:49,128:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 752
2023-10-12 17:11:49,129:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Thu, 12 Oct 2023 17:11:49 GMT
Content-Type: application/json
Content-Length: 752
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "Q5kkPCShlRQ": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "renewalInfo": "https://acme-v02.api.letsencrypt.org/draft-ietf-acme-ari-01/renewalInfo/",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
}
2023-10-12 17:11:49,201:DEBUG:certbot._internal.storage:Should renew, less than 30 days before certificate expiry 2023-10-10 11:53:51 UTC.
2023-10-12 17:11:49,201:INFO:certbot._internal.renewal:Certificate is due for renewal, auto-renewing...
2023-10-12 17:11:49,202:DEBUG:certbot._internal.display.obj:Notifying user: Renewing an existing certificate for *.mydomain.com
2023-10-12 17:11:49,223:DEBUG:acme.client:Requesting fresh nonce
2023-10-12 17:11:49,223:DEBUG:acme.client:Sending HEAD request to https://acme-v02.api.letsencrypt.org/acme/new-nonce.
2023-10-12 17:11:49,381:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "HEAD /acme/new-nonce HTTP/1.1" 200 0
2023-10-12 17:11:49,382:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Thu, 12 Oct 2023 17:11:49 GMT
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: _s_5u1NQesnT8oZjPBpqAaEhMw2v6CydHCwhnmR5XInLyyHRIPE
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800


2023-10-12 17:11:49,382:DEBUG:acme.client:Storing nonce: _s_5u1NQesnT8oZjPBpqAaEhMw2v6CydHCwhnmR5XInLyyHRIPE
2023-10-12 17:11:49,383:DEBUG:acme.client:JWS payload:
b'{\n  "identifiers": [\n    {\n      "type": "dns",\n      "value": "*.mydomain.com"\n    }\n  ]\n}'
2023-10-12 17:11:49,386:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/new-order:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvNzI4MDE1OTk3IiwgIm5vbmNlIjogIl9zXzV1MU5RZXNuVDhvWmpQQnBxQWFFaE13MnY2Q3lkSEN3aG5tUjVYSW5MeXlIUklQRSIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvbmV3LW9yZGVyIn0",
  "signature": "Jj0Sp2hzhdI2hDnBEFcqvMhhIf0mPsH89YUFw-7Se-rglCtsxrxrituJzMpx2ES2FiMiCJYu-f8DKtVxM2hYSbxpho0Uhn0eWtmIUjkHZvZZs54cYQR7CLP2hZ37mn9eq-Jlb1GFLPTEqbgIUhel90RM09mLPEa2O-sq0S9RamdDlAHGqd3EpHGUQG4VqMxXYD7SGRMkQA2fLMJgoh1wdrWOKKfU_UbdXF-qG4DyBWSjSAc2HzXMRSRoc-6xIcXO2mjFz4jG9mA3fFT7WCCKf2MzPpqyeWyIBY1a7nK0VkQrvqjoa3WsHXx0XTMZUwUBST2raKGKUdnMwu4R8dqx4g",
  "payload": "ewogICJpZGVudGlmaWVycyI6IFsKICAgIHsKICAgICAgInR5cGUiOiAiZG5zIiwKICAgICAgInZhbHVlIjogIioubWd1eWFyZC5jb20iCiAgICB9CiAgXQp9"
}
2023-10-12 17:11:49,565:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/new-order HTTP/1.1" 201 338
2023-10-12 17:11:49,566:DEBUG:acme.client:Received response:
HTTP 201
Server: nginx
Date: Thu, 12 Oct 2023 17:11:49 GMT
Content-Type: application/json
Content-Length: 338
Connection: keep-alive
Boulder-Requester: 728015997
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Location: https://acme-v02.api.letsencrypt.org/acme/order/728015997/214563998906
Replay-Nonce: _s_5u1NQdwW67X0xzS5_dSAlKHRVyVeyJKM_Zg4ZROl40Auts9w
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "status": "pending",
  "expires": "2023-10-19T11:11:26Z",
  "identifiers": [
    {
      "type": "dns",
      "value": "*.mydomain.com"
    }
  ],
  "authorizations": [
    "https://acme-v02.api.letsencrypt.org/acme/authz-v3/273135072886"
  ],
  "finalize": "https://acme-v02.api.letsencrypt.org/acme/finalize/728015997/214563998906"
}
2023-10-12 17:11:49,567:DEBUG:acme.client:Storing nonce: _s_5u1NQdwW67X0xzS5_dSAlKHRVyVeyJKM_Zg4ZROl40Auts9w
2023-10-12 17:11:49,568:DEBUG:acme.client:JWS payload:
b''
2023-10-12 17:11:49,571:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/273135072886:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvNzI4MDE1OTk3IiwgIm5vbmNlIjogIl9zXzV1MU5RZHdXNjdYMHh6UzVfZFNBbEtIUlZ5VmV5SktNX1pnNFpST2w0MEF1dHM5dyIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYXV0aHotdjMvMjczMTM1MDcyODg2In0",
  "signature": "Xd5zPjUSbV039jIfs_bOBd2bajp8iiAQoKaRWLzZHkSMvbnTRliGzedTaQo9b01qiI-J1USTkbqdlXq15RujAhh-CdmD70qCVGup8-K5R-gfWsCBl5DwmUdA5ytGmRhbQCLSZt40OaNvAlfD3HwLeeAwg43yR7UcHfz1Mk-Pa4Lc-70TxaBImNlVeMM5G4UbR0Bb03o10OW8_krdXPz-NYssLiIDV9T-_sfP7zB_D2CJ54o2zHx5oeM3DWnHCQCprcbPlGlgi4IVzr3hhhq_AkpxI6IIvl_F9ziKiYI2-DecfPbSVt9sczs5wrb4wwOx7kOKWTVkwwZjD_46JAVjlg",
  "payload": ""
}
2023-10-12 17:11:49,732:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/273135072886 HTTP/1.1" 200 385
2023-10-12 17:11:49,733:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Thu, 12 Oct 2023 17:11:49 GMT
Content-Type: application/json
Content-Length: 385
Connection: keep-alive
Boulder-Requester: 728015997
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: _s_5u1NQSMtSKKLPhg0YW4NUXu-3WtIhNW_K3-1HP9ItVKD_ItQ
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "mydomain.com"
  },
  "status": "pending",
  "expires": "2023-10-19T11:11:26Z",
  "challenges": [
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/273135072886/g5IbYA",
      "token": "glIZphzibloH4KOTCCnIEVniRVcNdyghsHABrcOhIM8"
    }
  ],
  "wildcard": true
}
2023-10-12 17:11:49,734:DEBUG:acme.client:Storing nonce: _s_5u1NQSMtSKKLPhg0YW4NUXu-3WtIhNW_K3-1HP9ItVKD_ItQ
2023-10-12 17:11:49,734:INFO:certbot._internal.auth_handler:Performing the following challenges:
2023-10-12 17:11:49,735:INFO:certbot._internal.auth_handler:dns-01 challenge for mydomain.com
2023-10-12 17:11:49,786:DEBUG:filelock:Attempting to acquire lock 281473097129776 on /root/.lexicon_tld_set/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2023-10-12 17:11:49,787:DEBUG:filelock:Lock 281473097129776 acquired on /root/.lexicon_tld_set/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2023-10-12 17:11:49,788:DEBUG:filelock:Attempting to acquire lock 281473097131936 on /root/.lexicon_tld_set/urls/62bf135d1c2f3d4db4228b9ecaf507a2.tldextract.json.lock
2023-10-12 17:11:49,788:DEBUG:filelock:Lock 281473097131936 acquired on /root/.lexicon_tld_set/urls/62bf135d1c2f3d4db4228b9ecaf507a2.tldextract.json.lock
2023-10-12 17:11:49,789:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): publicsuffix.org:443
2023-10-12 17:11:50,376:DEBUG:urllib3.connectionpool:https://publicsuffix.org:443 "GET /list/public_suffix_list.dat HTTP/1.1" 200 81964
2023-10-12 17:11:50,404:DEBUG:filelock:Attempting to release lock 281473097131936 on /root/.lexicon_tld_set/urls/62bf135d1c2f3d4db4228b9ecaf507a2.tldextract.json.lock
2023-10-12 17:11:50,405:DEBUG:filelock:Lock 281473097131936 released on /root/.lexicon_tld_set/urls/62bf135d1c2f3d4db4228b9ecaf507a2.tldextract.json.lock
2023-10-12 17:11:50,421:DEBUG:filelock:Attempting to release lock 281473097129776 on /root/.lexicon_tld_set/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2023-10-12 17:11:50,422:DEBUG:filelock:Lock 281473097129776 released on /root/.lexicon_tld_set/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2023-10-12 17:11:50,458:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): eu.api.ovh.com:443
2023-10-12 17:11:50,730:DEBUG:urllib3.connectionpool:https://eu.api.ovh.com:443 "GET /1.0/auth/time HTTP/1.1" 200 10
2023-10-12 17:11:50,779:DEBUG:urllib3.connectionpool:https://eu.api.ovh.com:443 "GET /1.0/domain/zone/ HTTP/1.1" 200 None
2023-10-12 17:11:50,901:DEBUG:urllib3.connectionpool:https://eu.api.ovh.com:443 "GET /1.0/domain/zone/mydomain.com/status HTTP/1.1" 200 None
2023-10-12 17:11:50,942:DEBUG:filelock:Attempting to acquire lock 281473084944640 on /root/.lexicon_tld_set/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2023-10-12 17:11:50,943:DEBUG:filelock:Lock 281473084944640 acquired on /root/.lexicon_tld_set/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2023-10-12 17:11:50,944:DEBUG:filelock:Attempting to release lock 281473084944640 on /root/.lexicon_tld_set/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2023-10-12 17:11:50,944:DEBUG:filelock:Lock 281473084944640 released on /root/.lexicon_tld_set/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2023-10-12 17:11:50,961:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): eu.api.ovh.com:443
2023-10-12 17:11:51,229:DEBUG:urllib3.connectionpool:https://eu.api.ovh.com:443 "GET /1.0/auth/time HTTP/1.1" 200 10
2023-10-12 17:11:51,264:DEBUG:urllib3.connectionpool:https://eu.api.ovh.com:443 "GET /1.0/domain/zone/ HTTP/1.1" 200 None
2023-10-12 17:11:51,339:DEBUG:urllib3.connectionpool:https://eu.api.ovh.com:443 "GET /1.0/domain/zone/mydomain.com/status HTTP/1.1" 200 None
2023-10-12 17:11:51,394:DEBUG:urllib3.connectionpool:https://eu.api.ovh.com:443 "GET /1.0/domain/zone/mydomain.com/record?fieldType=TXT&subDomain=_acme-challenge HTTP/1.1" 200 2
2023-10-12 17:11:51,396:DEBUG:lexicon._private.providers.ovh:list_records: []
2023-10-12 17:11:51,544:DEBUG:urllib3.connectionpool:https://eu.api.ovh.com:443 "POST /1.0/domain/zone/mydomain.com/record HTTP/1.1" 200 None
2023-10-12 17:11:51,652:DEBUG:urllib3.connectionpool:https://eu.api.ovh.com:443 "POST /1.0/domain/zone/mydomain.com/refresh HTTP/1.1" 204 0
2023-10-12 17:11:51,654:DEBUG:certbot.plugins.dns_common_lexicon:Encountered error adding TXT record: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/certbot/src/certbot/certbot/plugins/dns_common_lexicon.py", line 221, in _perform
    operations.create_record(rtype='TXT', name=validation_name, content=validation)
  File "/usr/local/lib/python3.10/site-packages/lexicon/client.py", line 34, in create_record
    return self.provider.create_record(rtype, name, content)
  File "/usr/local/lib/python3.10/site-packages/lexicon/_private/providers/ovh.py", line 129, in create_record
    self._post(f"/domain/zone/{domain}/refresh")
  File "/usr/local/lib/python3.10/site-packages/lexicon/interfaces.py", line 168, in _post
    return self._request("POST", url, data=data, query_params=query_params)
  File "/usr/local/lib/python3.10/site-packages/lexicon/_private/providers/ovh.py", line 260, in _request
    return result.json()
  File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2023-10-12 17:11:51,658:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/certbot/src/certbot/certbot/plugins/dns_common_lexicon.py", line 221, in _perform
    operations.create_record(rtype='TXT', name=validation_name, content=validation)
  File "/usr/local/lib/python3.10/site-packages/lexicon/client.py", line 34, in create_record
    return self.provider.create_record(rtype, name, content)
  File "/usr/local/lib/python3.10/site-packages/lexicon/_private/providers/ovh.py", line 129, in create_record
    self._post(f"/domain/zone/{domain}/refresh")
  File "/usr/local/lib/python3.10/site-packages/lexicon/interfaces.py", line 168, in _post
    return self._request("POST", url, data=data, query_params=query_params)
  File "/usr/local/lib/python3.10/site-packages/lexicon/_private/providers/ovh.py", line 260, in _request
    return result.json()
  File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/certbot/src/certbot/certbot/_internal/auth_handler.py", line 88, in handle_authorizations
    resps = self.auth.perform(achalls)
  File "/opt/certbot/src/certbot/certbot/plugins/dns_common.py", line 76, in perform
    self._perform(domain, validation_domain_name, validation)
  File "/opt/certbot/src/certbot/certbot/plugins/dns_common_lexicon.py", line 224, in _perform
    raise errors.PluginError('Error adding TXT record: {0}'.format(e))
certbot.errors.PluginError: Error adding TXT record: Expecting value: line 1 column 1 (char 0)

2023-10-12 17:11:51,659:DEBUG:certbot._internal.error_handler:Calling registered functions
2023-10-12 17:11:51,659:INFO:certbot._internal.auth_handler:Cleaning up challenges
2023-10-12 17:11:51,695:DEBUG:filelock:Attempting to acquire lock 281473082809680 on /root/.lexicon_tld_set/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2023-10-12 17:11:51,696:DEBUG:filelock:Lock 281473082809680 acquired on /root/.lexicon_tld_set/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2023-10-12 17:11:51,697:DEBUG:filelock:Attempting to release lock 281473082809680 on /root/.lexicon_tld_set/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2023-10-12 17:11:51,697:DEBUG:filelock:Lock 281473082809680 released on /root/.lexicon_tld_set/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2023-10-12 17:11:51,720:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): eu.api.ovh.com:443
2023-10-12 17:11:52,034:DEBUG:urllib3.connectionpool:https://eu.api.ovh.com:443 "GET /1.0/auth/time HTTP/1.1" 200 10
2023-10-12 17:11:52,110:DEBUG:urllib3.connectionpool:https://eu.api.ovh.com:443 "GET /1.0/domain/zone/ HTTP/1.1" 200 None
2023-10-12 17:11:52,199:DEBUG:urllib3.connectionpool:https://eu.api.ovh.com:443 "GET /1.0/domain/zone/mydomain.com/status HTTP/1.1" 200 None
2023-10-12 17:11:52,242:DEBUG:filelock:Attempting to acquire lock 281473085303072 on /root/.lexicon_tld_set/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2023-10-12 17:11:52,243:DEBUG:filelock:Lock 281473085303072 acquired on /root/.lexicon_tld_set/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2023-10-12 17:11:52,244:DEBUG:filelock:Attempting to release lock 281473085303072 on /root/.lexicon_tld_set/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2023-10-12 17:11:52,244:DEBUG:filelock:Lock 281473085303072 released on /root/.lexicon_tld_set/publicsuffix.org-tlds/de84b5ca2167d4c83e38fb162f2e8738.tldextract.json.lock
2023-10-12 17:11:52,261:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): eu.api.ovh.com:443
2023-10-12 17:11:52,559:DEBUG:urllib3.connectionpool:https://eu.api.ovh.com:443 "GET /1.0/auth/time HTTP/1.1" 200 10
2023-10-12 17:11:52,618:DEBUG:urllib3.connectionpool:https://eu.api.ovh.com:443 "GET /1.0/domain/zone/ HTTP/1.1" 200 None
2023-10-12 17:11:52,689:DEBUG:urllib3.connectionpool:https://eu.api.ovh.com:443 "GET /1.0/domain/zone/mydomain.com/status HTTP/1.1" 200 None
2023-10-12 17:11:52,759:DEBUG:urllib3.connectionpool:https://eu.api.ovh.com:443 "GET /1.0/domain/zone/mydomain.com/record?fieldType=TXT&subDomain=_acme-challenge HTTP/1.1" 200 12
2023-10-12 17:11:52,812:DEBUG:urllib3.connectionpool:https://eu.api.ovh.com:443 "GET /1.0/domain/zone/mydomain.com/record/5288946926 HTTP/1.1" 200 None
2023-10-12 17:11:52,813:DEBUG:lexicon._private.providers.ovh:list_records: [{'type': 'TXT', 'name': '_acme-challenge.mydomain.com', 'ttl': 60, 'content': 'XUBy2LmYnK0XEIf5m3C55irRFyXmqQioIy_YcxCyy6E', 'id': 5288946926}]
2023-10-12 17:11:52,814:DEBUG:lexicon._private.providers.ovh:delete_records: [5288946926]
2023-10-12 17:11:52,936:DEBUG:urllib3.connectionpool:https://eu.api.ovh.com:443 "DELETE /1.0/domain/zone/mydomain.com/record/5288946926 HTTP/1.1" 200 4
2023-10-12 17:11:53,031:DEBUG:urllib3.connectionpool:https://eu.api.ovh.com:443 "POST /1.0/domain/zone/mydomain.com/refresh HTTP/1.1" 204 0
2023-10-12 17:11:53,033:DEBUG:certbot.plugins.dns_common_lexicon:Encountered error deleting TXT record: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/certbot/src/certbot/certbot/plugins/dns_common_lexicon.py", line 236, in _cleanup
    operations.delete_record(rtype='TXT', name=validation_name, content=validation)
  File "/usr/local/lib/python3.10/site-packages/lexicon/client.py", line 73, in delete_record
    return self.provider.delete_record(identifier, rtype, name, content)
  File "/usr/local/lib/python3.10/site-packages/lexicon/_private/providers/ovh.py", line 210, in delete_record
    self._post(f"/domain/zone/{domain}/refresh")
  File "/usr/local/lib/python3.10/site-packages/lexicon/interfaces.py", line 168, in _post
    return self._request("POST", url, data=data, query_params=query_params)
  File "/usr/local/lib/python3.10/site-packages/lexicon/_private/providers/ovh.py", line 260, in _request
    return result.json()
  File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2023-10-12 17:11:53,037:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/certbot/src/certbot/certbot/plugins/dns_common_lexicon.py", line 221, in _perform
    operations.create_record(rtype='TXT', name=validation_name, content=validation)
  File "/usr/local/lib/python3.10/site-packages/lexicon/client.py", line 34, in create_record
    return self.provider.create_record(rtype, name, content)
  File "/usr/local/lib/python3.10/site-packages/lexicon/_private/providers/ovh.py", line 129, in create_record
    self._post(f"/domain/zone/{domain}/refresh")
  File "/usr/local/lib/python3.10/site-packages/lexicon/interfaces.py", line 168, in _post
    return self._request("POST", url, data=data, query_params=query_params)
  File "/usr/local/lib/python3.10/site-packages/lexicon/_private/providers/ovh.py", line 260, in _request
    return result.json()
  File "/usr/local/lib/python3.10/site-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/certbot", line 33, in <module>
    sys.exit(load_entry_point('certbot', 'console_scripts', 'certbot')())
  File "/opt/certbot/src/certbot/certbot/main.py", line 19, in main
    return internal_main.main(cli_args)
  File "/opt/certbot/src/certbot/certbot/_internal/main.py", line 1873, in main
    return config.func(config, plugins)
  File "/opt/certbot/src/certbot/certbot/_internal/main.py", line 1600, in certonly
    lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
  File "/opt/certbot/src/certbot/certbot/_internal/main.py", line 131, in _get_and_save_cert
    renewal.renew_cert(config, domains, le_client, lineage)
  File "/opt/certbot/src/certbot/certbot/_internal/renewal.py", line 396, in renew_cert
    new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
  File "/opt/certbot/src/certbot/certbot/_internal/client.py", line 428, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/opt/certbot/src/certbot/certbot/_internal/client.py", line 496, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, self.config, best_effort)
  File "/opt/certbot/src/certbot/certbot/_internal/auth_handler.py", line 88, in handle_authorizations
    resps = self.auth.perform(achalls)
  File "/opt/certbot/src/certbot/certbot/plugins/dns_common.py", line 76, in perform
    self._perform(domain, validation_domain_name, validation)
  File "/opt/certbot/src/certbot/certbot/plugins/dns_common_lexicon.py", line 224, in _perform
    raise errors.PluginError('Error adding TXT record: {0}'.format(e))
certbot.errors.PluginError: Error adding TXT record: Expecting value: line 1 column 1 (char 0)
2023-10-12 17:11:53,039:ERROR:certbot._internal.log:Error adding TXT record: Expecting value: line 1 column 1 (char 0)

An issue is also open on certbot side but i think resolution need to be find on lexicon side : certbot/certbot#9799

Thanks for your help

[adferrand]

Why OVH, why ...

So basically the issue is due to an error raised by response.json() trying to extract an invalid JSON from the response body after a request to OVH APIs.

Three scenarios can happen with response.json():

1. the response body is effectively a JSON and response.json() returns a JSON properly parsed as a dict,
2. there is no response body and response.json() returns None (up to the caller to know that no response is expected and so no further processing is relevant),
3. the response body exists and is not a JSON, and response.json() raises a requests.exceptions.JSONDecodeError.

Documentation of OVH has always stated that API response body was a JSON for some POST request, and no body for others (typically /refresh). So the code was designed to handle 1) and 2) since 3) should be considered as "something went wrong".

However it seems that now, OVH APIs may return an empty body instead of no body sometimes. I really mean sometimes here because the behavior appears to be flaky. I tested creation and deletion of records: sometimes no body is returned for /refresh for instance, and everything goes fine; sometimes an empty body is returned for the same endpoint (/refresh) and the code crashes.

I released a new version of Lexicon that explicitly catches 3) to return None instead of erroring out.

Release 3.15.1 is on its way.

[trnsnt]

Hello,

Api has been changed to return a HTTP 204 instead of an HTTP 200 with body = null
I think, you should not try to call .json when status code is 204 as by definition HTTP 204 means no content !

[adferrand]

The code was already completely fine with either 200 or 204 code, and also fine with a response without body. In this case response.json() will return None.

The problem here was that some APIs started to return an empty body, for which response.json() cannot work given that an empty string is not a valid JSON.

What is worse is that the same endpoint returns sometimes no body, sometimes an empty body. I guess that it is related to some internal state on OVH side, like the endpoint triggers an action or not. But this is not specified anywhere in the doc, that is stated only that no body is returned for this kind of endpoints.

[camille-sound4]

Hello,

Thanks for the fast correction !
I had the same problem, and the correction worked for the Error adding TXT record with certbot 2.7.2 which is using lexicon 3.15.1.

However, I still have another error Zone mydomain.com is not deployed happening since last week, which is probably related :

Performing the following challenges:
dns-01 challenge for mysite.mydomain.com
Attempting to acquire lock 654219847864646 on /root/.lexicon_tld_set/publicsuffix.org-tlds/da645388e42c4fc4b1dfeeec3ac86334.tldextract.json.lock
Lock 654219847864646 acquired on /root/.lexicon_tld_set/publicsuffix.org-tlds/da645388e42c4fc4b1dfeeec3ac86334.tldextract.json.lock
Attempting to release lock 654219847864646 on /root/.lexicon_tld_set/publicsuffix.org-tlds/da645388e42c4fc4b1dfeeec3ac86334.tldextract.json.lock
Lock 654219847864646 released on /root/.lexicon_tld_set/publicsuffix.org-tlds/da645388e42c4fc4b1dfeeec3ac86334.tldextract.json.lock
Starting new HTTPS connection (1): eu.api.ovh.com:443
https://eu.api.ovh.com:443 "GET /1.0/auth/time HTTP/1.1" 200 10
https://eu.api.ovh.com:443 "GET /1.0/domain/zone/ HTTP/1.1" 200 None
https://eu.api.ovh.com:443 "GET /1.0/domain/zone/mydomain.com/status HTTP/1.1" 200 None
Encountered exception:
Traceback (most recent call last):
  File "/snap/certbot/3420/lib/python3.8/site-packages/certbot/plugins/dns_common_lexicon.py", line 246, in _resolve_domain
    with Client(self._build_lexicon_config(domain_name)):
  File "/snap/certbot-dns-ovh/current/lib/python3.8/site-packages/lexicon/client.py", line 151, in __enter__
    raise e
  File "/snap/certbot-dns-ovh/current/lib/python3.8/site-packages/lexicon/client.py", line 144, in __enter__
    provider.authenticate()
  File "/snap/certbot-dns-ovh/current/lib/python3.8/site-packages/lexicon/_private/providers/ovh.py", line 96, in authenticate
    raise AuthenticationError(f"Zone {domain} is not deployed")
lexicon.exceptions.AuthenticationError: Zone mydomain.com is not deployed

Thanks for your help

[e-gautier]

Hi,

On behalf of the OVH DNS team I apologize for the trouble.
We migrated the domain API on another platform lately and this meant to be transparent (well...mostly).
We made the choice to switch the 200 statuses (body null) by 204 statuses (w/o content) and didn't expect this to have any impact.
The fickleness you encountered was mostly due to the API going through our blue/green migration process.
A communication will be made on the mailing list and the documentation will be updated as soon as possible.

[adferrand]

Thanks a lot @e-gautier for you response. It clarifies the situation.

I think that changing from HTTP 200 to HTTP 204 in these endpoints makes sense. Alone this change would have had no impact, at least on Lexicon/Certbot side, and I bet in most other places also. Indeed the semantic usually applied is given by calling some abstraction method to check that the response is "valid", and any HTTP client will consider that 200 and 204 are valid. Here we are calling response.raise_for_status() and it follows this semantic.

The real problem is that these endpoints are now returning a body, which is an empty string. This is a direct violation of the intention of HTTP 204 status, and also of any response containing the header Content-Type: application/json since the body is not JSON parseable. You could argue here that no body is also problematic with this Content-Type, but I would say that it would be the case with any Content-Type, and also that not returning any body when there is no data to return is a sufficiently common approach in APIs to consider that any HTTP client implementation is very likely to treat correctly this case. For instance response.json() will not blow up and will just return None.

So I would strongly advise you to stop the API from returning a body when HTTP 204 is involved. Not only that it will kill right now all errors encountered by OVH users with Lexicon and Certbot, but you get a clean implementation that is future-proof.

[e-gautier]

Hello @adferrand,
I agree about the Content-Type header.
What do you mean by these endpoints are now returning a body, which is an empty string? Afaik these endpoints are returning a new line right after the headers which is compliant to the RFC and the .json() method will always unconditionally try to parse JSON from the request content, empty or not.

[adferrand]

You are right and I interpretated wrongly the situation.

Before these APIs (and still a couple of others, like DELETE /domain/zone/{DOMAIN}/record/{RECORD}, were returning HTTP 200 with a body composed of null + carriage return, that JSON parsers interpret as the "null" value as appropriate for the target language (so None in Python) like the built-in module json (reference: https://docs.python.org/3/library/json.html#encoders-and-decoders), consumed by request to process JSON bodies.

In that extent HTTP 204 makes an actual body not necessary, even null. RFC is stating to just return a new line in the HTTP package, as these APIs do now. This is the unfortunate consequence in HTTP APIs of trying to express a null value with a text-based protocol, and here it implies to move from an actual body to no body, and this is not backward compatible.

So I concur with this change in the long term, but sadly it still represents a move in the API in short term that would deserve at least a migration path and potentially some versioning on the API...

[dauphinpasdroit]

Hello guys,

Does the issue has been fixed lately ?

[arcameca]

Yes, it has been fixed.