pdns: API endpoint not at URL root resulting in incorrect URL queried and thus failing with error code 404

[jotasi]

Welcome

• Yes, I'm using a binary release within 2 latest releases.
• Yes, I've searched similar issues on GitHub and didn't find any.
• Yes, I've included all information below (version, config, etc).

What did you expect to see?

When requesting a first or renewing an existing certificate via DNS challenge and PowerDNS API with the API endpoint not being located at the URL root (e.g., https://login.udmedia.de/dns/api instead of https://login.udmedia.de/api), the command should run through without an error.

What did you see instead?

When requesting a first or renewing an existing certificate via DNS challenge and PDNS API with the API endpoint not being located at the URL root (e.g., https://login.udmedia.de/dns/api instead of https://login.udmedia.de/api), the command fails with an error code 404, indicating that an unknown URL was queried.

After debugging, this was pinpointed to the URL generated as endpoint for updating within UpdateRecords in providers/dns/pdns/internal/client.go not being the expected https://login.udmedia.de/dns/api/v1/servers/udmedia/zones/example.com. but instead https://login.udmedia.de/dns/api/v1/dns/api/v1/servers/udmedia/zones/example.com. (i.e., with duplicated dns/api/v1/).

The reason for that is that the URL of the host to which the path is appended by joinPath is already containing the /dns but the URL of the zone that is returned by the provider's PDNS compatible API (determined via GetHostedZone) and AFAIU also by PowerDNS itself (tentative as I don't have a PowerDNS installation to try this out with but see, e.g., discussion here) is an absolute URL path also containing the starting /dns. Therefore, /dns of the Host URL is extended by a second /dns from the Zone URL. Furthermore, in joinPath for API version != 0, any path to add that is not starting with /api (which is the case here as the path starts with /dns/api/...) is also prepended by /api/v1. This then results in the superfluous /api/v1/dns being added between the host URL and the Zone URL path.

This could be fixed by adding a second join function (e.g., joinAbsolutePath) that removes any remaining path from the Host URL before joining and (as this seems unnecessary in that case to me, but please correct me there if this is actually needed for, e.g., earlier version of PowerDNS or the API) not trying to guess whether /api/v1 needs to be prepended to the URL path. The function could look somewhat like this:

func (c *Client) joinAbsolutePath(elem ...string) *url.URL {
	p := path.Join(elem...)

	rawHost := *c.Host
	rawHost.Path = ""

	return rawHost.JoinPath(p)
}

and could replace joinPath here and here.

I can confirm that this fixes the issue at least for my provider. If helpful, I can provide a corresponding PR also including some tests of the new function. However, the caveat here is that I have limited experience with go and PowerDNS in general and don't want to inadvertently break something for someone else.

How do you use lego?

Binary

Reproduction steps

Try requesting a first or renewing an existing certificate via DNS challenge and PDNS API (API version v1) with the API endpoint not being located at the URL root (PDNS_API_URL containing a non-empty path, e.g., https://login.udmedia.de/dns instead of https://login.udmedia.de for an API endpoint at https://login.udmedia.de/dns/api instead of https://login.udmedia.de/api).

The command used (and environment variables set (except for the API key), see Logs section.

(See also discussion #2122)

For my understanding of the origin of the error after debugging this, see section on "What did you see instead?".

Version of lego

lego version 4.15.0 linux/386

Logs

Command executed to generate the log:

export PDNS_API_URL=https://login.udmedia.de/dns
export PDNS_SERVER_NAME=udmedia
export PDNS_API_KEY=<read_from_file>
export PDNS_API_VERSION=1  # Required as this provider returns not a list but just a single entry when querying the version
./lego --accept-tos --path '.' -d 'some_domain.de' --email 'some_email@example.xom' --key-type 'ec256' --dns 'pdns' --dns.resolvers '5.1.66.255:53' --server 'https://acme-staging-v02.api.letsencrypt.org/directory' run

2024/02/28 21:46:31 [INFO] [some_domain.de] acme: Obtaining bundled SAN certificate
2024/02/28 21:46:31 [INFO] [some_domain.de AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/<snip>
2024/02/28 21:46:31 [INFO] [some_domain.de] acme: Could not find solver for: tls-alpn-01
2024/02/28 21:46:31 [INFO] [some_domain.de] acme: Could not find solver for: http-01
2024/02/28 21:46:31 [INFO] [some_domain.de] acme: use dns-01 solver
2024/02/28 21:46:31 [INFO] [some_domain.de] acme: Preparing to solve DNS-01
2024/02/28 21:46:32 [INFO] [some_domain.de] acme: Cleaning DNS-01 challenge
2024/02/28 21:46:32 [WARN] [some_domain.de] acme: cleaning up failed: pdns: unexpected status code: [status code: 404] body: {
    "error": "Not found"
}
2024/02/28 21:46:32 [INFO] Deactivating auth: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/<snip>
2024/02/28 21:46:32 Could not obtain certificates:
	error: one or more domains had a problem:
[some_domain.de] [some_domain.de] acme: error presenting token: pdns: unexpected status code: [status code: 404] body: {
    "error": "Not found"
}

Go environment (if applicable)

$ go version && go env
# paste output here

[ldez]

Hello,

I'm not sure to understand your problem.

As you can see we have dedicated tests on joinPath:

lego/providers/dns/pdns/internal/client_test.go

Line 66 in 82e9a5e

func TestClient_joinPath(t *testing.T) {

Your implementation will break everything.

And the tests seems to say the opposite of the behavior you described 🤔 .

lego/providers/dns/pdns/internal/client_test.go

Lines 74 to 80 in 82e9a5e

	{
	desc: "host with path",
	apiVersion: 1,
	baseURL: "https://example.com/test",
	uri: "/foo",
	expected: "https://example.com/test/api/v1/foo",
	},

[jotasi]

Hi,

Thanks for the quick reply!

Please find a bit more concrete explanation below.

There are four places where joinPath is used (which just adds the path to the PDNS_API_URL, potentially adding /api/v1 to the front if the provided path does not start with /api and if the API version is not 0):

With a relative path (well represented by the tests and functional in my use case):

lego/providers/dns/pdns/internal/client.go

Lines 55 to 58 in 82e9a5e

	func (c *Client) getAPIVersion(ctx context.Context) (int, error) {
	endpoint := c.joinPath("/", "api")
	req, err := newJSONRequest(ctx, http.MethodGet, endpoint, nil)

With a relative path (well represented by the tests and functional in my use case):

lego/providers/dns/pdns/internal/client.go

Lines 84 to 87 in 82e9a5e

	func (c *Client) GetHostedZone(ctx context.Context, authZone string) (*HostedZone, error) {
	endpoint := c.joinPath("/", "servers", c.serverName, "zones", dns.Fqdn(authZone))
	req, err := newJSONRequest(ctx, http.MethodGet, endpoint, nil)

With an absolute path (not represented by the tests and not functional in my use case):

lego/providers/dns/pdns/internal/client.go

Lines 119 to 122 in 82e9a5e

	func (c *Client) UpdateRecords(ctx context.Context, zone *HostedZone, sets RRSets) error {
	endpoint := c.joinPath("/", zone.URL)
	req, err := newJSONRequest(ctx, http.MethodPatch, endpoint, sets)

With an absolute path (not represented by the tests and not functional in my use case):

lego/providers/dns/pdns/internal/client.go

Lines 135 to 140 in 82e9a5e

	func (c *Client) Notify(ctx context.Context, zone *HostedZone) error {
	if c.apiVersion < 1 || zone.Kind != "Master" && zone.Kind != "Slave" {
	return nil
	}
	endpoint := c.joinPath("/", zone.URL, "/notify")

Only the first two are used with a ("manually" built) path that is relative to the APIs root path (/api and /servers<serverName>/zones/<authZone>. These are represented well in, e.g., the test case(s) you have included above and also work for my use case.

The problem is with the latter two usages. Here, the path that is passed to joinPath is starting with zone.URL, where zone is the parsed return of the API call within GetHostedZone (see second code block above). However, at least for my provider, the URL that is returned by querying, in my case, https://login.udmedia.de/dns/api/v1/servers/udmedia/zones/example.com. is an absolute path to the host's root and thus also includes the relative path to the API again that is aready part of PDNS_API_URL (i.e., starting with /dns below rather than /api/v1/...):

{
    "id": "example.com.",
    "url": "\/dns\/api\/v1\/servers\/udmedia\/zones\/example.com.",
    "name": "example.com.",
    "type": "Zone",
<snip>

When this path is then passed to joinPath, this leads to URL with the duplicated /dns/api/v1/dns/api/v1 that I have listed in my original post.

My crude solution suggested above for the 3rd and 4th usage above (NOT replacing joinPath for the first two) works for my case and should also work for the standard case, where the API endpoint sits at the URL root with API version 1 where the /api/v1 should already be part of zone.URL.

However, there are also possibly less invasive solutions. If any of them sounds reasonable, please let me know and I can potentially flesh them out some more:

• Instead of using zone.URL in the 3rd and 4th joinPath calls above, one could recreated the relative URL analogously to the 2nd usage. With a relative URL, this should then work fine. From reading the PowerDNS API documentation, the URL sounds to be fixed and thus zone.URL usage just saves some work but should hopefully be replaceable.
• When zone.URL should still be used, an environmental variable option (something like PDNS_ZONE_URL_ABSOLUTE=1), whether it is known to contain the absolute URL to the URL's root, could be added and only if that is set, one could basically use my crude solution or try to retrospectively make the zone.URL a relative path before using joinPath by removing the path from PDNS_API_URL from the start of the zone.URL.