"plugin/auto: no next plugin found" causing apparently erroneous NOERROR logs

[josh-fayer-cleandns]

Hi there! I wasn't sure whether to submit this as a bug report or a discussion question, so I figured I'd start here so as to create less clutter. If you think this represents a genuine bug, I'd be happy to rewrite it with some additional detail as an Issue.

We have a CoreDNS server hosting zonefiles using the auto plugin to search in a directory like so

com { 
    auto { 
        directory /zones/com
        reload 60s
    } 
    log
    errors
    dnstap tcp://pdns-collector:5999 full
}

On the clientside, this appears to work fine — zonefiles we have are served correctly, and when there is no zonefile for a requested domain it results in a SERVFAIL. However, when there is no zonefile, the auto plugin throws an error and returns no rcode or data and CoreDNS logs this effectively as an empty response:

from dig client:

dig @localhost +tcp -p 1053 example.com

; <<>> DiG 9.10.6 <<>> @localhost +tcp -p 1053 example.com
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 52788
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

in CoreDNS logs:

[INFO] <ipaddr>:<port> - 52788 "A IN example.com. tcp 40 false 65535" - - 0 0.000551542s
[ERROR] plugin/errors: 2 example.com. A: plugin/auto: no next plugin found

CoreDNS's behavior, as described in the docs, is that "a nodata response sets the return code to NOERROR", so our dnstap logs reflect such a query as a NOERROR. This makes it difficult to distinguish such responses from genuine NOERROR responses served from our zonefiles, and causes a difference between the response code served to the client (SERVFAIL) and the response code recorded in our logs (NOERROR). Ideally, we want to be able to filter out SERVFAILs in our dnstap collector and only record logs for domains we host.

Attempts to resolve this by adding an additional plugin below the auto plugin to resolve the "no next plugin" error haven't so far been successful.

For example, a simple solution might be to add a template that returns a SERVFAIL for all domains below the auto plugin, i.e.

com:1053 { 
    auto { 
        directory /zones/com
        reload 60s
    }
    template ANY ANY { 
        rcode SERVFAIL
    }
}

however this does not work as intended and simply returns all queries as SERVFAILs. Likewise, the auto plugin appears not to support the fallthrough directive so we cannot fall this through to a separate block that can handle and return SERVFAILs.

As best I can tell, to resolve this issue, we would need to do one of three things:

1. modify the default behavior of CoreDNS to make no-data responses instead result in a SERVFAIL
2. fallthrough no-data responses from the auto plugin to another server block that can use a template plugin to respond all queries as SERVFAILs (without overriding the response from the auto plugin)
3. modify the response from the auto plugin to return a SERVFAIL rather than no-rcode when a domain cannot be found

I haven't been able to find any documentation suggesting any of these are possible.

Is there something I'm missing about the way plugin chains work, or a configuration option I've skipped over that might help resolve this problem?

Thanks for any help you can provide!

[chrisohaver]

However, when there is no zonefile, the auto plugin throws an error and returns no rcode or data and CoreDNS logs this effectively as an empty response:
from dig client:

dig @localhost +tcp -p 1053 example.com

; <<>> DiG 9.10.6 <<>> @localhost +tcp -p 1053 example.com
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 52788
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

The response shows a SERVFAIL. SERVFAIL is a response code. This is not a NOERROR response code.

CoreDNS's behavior, as described in the docs, is that "a nodata response sets the return code to NOERROR", so our dnstap logs reflect such a query as a NOERROR.

This is a standard DNS thing, not a CoreDNS thing. "nodata" or "NODATA" is a pseudo response code that is actually represented in the DNS protocol as NOERROR response code with an empty answer section. People use the "NODATA" shorthand because it is shorter than saying "NOERROR-with-an-empty-answer-section."
The SERVFAIL response code is unrelated. It's used when a proper response cannot be formed - usually due to internal failure or server misconfiguration.

For example, a simple solution might be to add a template that returns a SERVFAIL for all domains below the auto plugin, i.e.
...
however this does not work as intended and simply returns all queries as SERVFAILs. Likewise, the auto plugin appears not to support the fallthrough directive so we cannot fall this through to a separate block that can handle and return SERVFAILs.

That doesn't work because order of plugin execution is statically defined at compile time - the order that you list different plugins in the configuration does not matter. The template plugin always executes before the auto plugin.

As best I can tell, to resolve this issue, we would need to do one of three things:

• modify the default behavior of CoreDNS to make no-data responses instead result in a SERVFAIL

Per your example, it does this already.

[chrisohaver]

Im a bit unclear on what the issue is. Perhaps if you rephrase this as "what you want to do" rather than "describing the problem" it would be clearer?

[chrisohaver]

Oh - I think I see... this issue is with the log entry ...

[INFO] <ipaddr>:<port> - 52788 "A IN example.com. tcp 40 false 65535" - - 0 0.000551542s

which falsely logs the response code as 0, when the response was actually 2 (a SERVFAIL).

yes, this appears to be a bug.

[chrisohaver]

Correction ... code is not a 0, rather a "-" which represents a nil value, not a zero.

[INFO] <ipaddr>:<port> - 52788 "A IN example.com. tcp 40 false 65535" - - 0 0.000551542s
                                                         this dash ---^

It is a bug, but not sure if it's an easy fix due to some difficult-to-explain shenanigans that the plugins and dns server play in tandem when handling SERVFAIL responses.

[chrisohaver]

A hacky workaround would be to forward non-existent zones to a local listener that answers with SERVFAIL.

com:1053 { 
    auto { 
        directory /zones/com
        reload 60s
    }
    forward . 127.0.0.1:1054
}

com:1054 { 
    template ANY ANY { 
        rcode SERVFAIL
    }
}

[josh-fayer-cleandns]

This is a very clever workaround! Didn't consider forwarding it to another port locally. Marking this as the accepted answer for anyone who runs into the same issue, and I'll submit this as an Issue with links to this conversation.

[josh-fayer-cleandns]

Worth noting for the purposes of investigating this: with the forwarding solution implemented, I am now seeing stdout logs from the log plugin responding correctly that those are SERVFAILs.

[INFO] 127.0.0.1:58574 - 62430 "A IN example.com. tcp 38 false 65535" SERVFAIL qr,aa 38 0.001495083s

However, dnstap is still sending logs as though they're NOERRORs. To confirm this, I have the go-dnscollector package listening for tap messages and simply logging them to the stdout:

CLIENT_QUERY NOERROR 127.0.0.1 58574 IPv4 TCP 38b example.com A 0.000000