SERVFAIL handling with --all-servers

[taam]

Currently with --all-servers enabled, it seems like the fastest response is forwarded, even if that happens to be a SERVFAIL, which is pretty bad for some use cases. A SERVFAIL can indicate network problems, e.g. some DNS servers/proxies answer very quickly with SERVFAIL, when their own upstream is dead (which I guess is correct and reasonable), in this case --all-servers makes things even worse than using just a single server, because now if any upstream is dead, we're always failing.

I suggest changing SERVFAIL handling to by default only forward it, if all upstream servers happen to respond with SERVFAIL.

In case someone brings DNSSEC as argument, maybe SERVFAILs can optionally be forwarded immediately, if coming with an appropriate EDE error code (RFC 8914).

(The SERVFAIL handling in fallback mode might be related: #257)

[ankino17]

I also found same issue, tested on Hong Kong server

2023/10/27 16:03:47 539786#21 [debug] https://149.112.112.10:443/dns-query: response received over tcp: ok
2023/10/27 16:03:47 539786#21 [debug] upstream https://149.112.112.10:443/dns-query exchanged ;www.arlingtoncemetery.mil.	IN	 A successfully in 73.184915ms
2023/10/27 16:03:47 539786#10 [debug] proxy: replying from upstream: rtt is 73.826264ms
2023/10/27 16:03:47 539786#10 [debug] github.com/AdguardTeam/dnsproxy/proxy.(*Proxy).logDNSMessage(): OUT: ;; opcode: QUERY, status: SERVFAIL, id: 36545
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version 0; flags:; udp: 4096

;; QUESTION SECTION:
;www.arlingtoncemetery.mil.	IN	 A

2023/10/27 16:03:47 539786#20 [debug] https://9.9.9.10:443/dns-query: response received over tcp: ok
2023/10/27 16:03:47 539786#20 [debug] upstream https://9.9.9.10:443/dns-query exchanged ;www.arlingtoncemetery.mil.	IN	 A successfully in 133.636366ms
2023/10/27 16:03:47 539786#19 [debug] https://208.67.220.2:443/dns-query: response received over tcp: ok
2023/10/27 16:03:47 539786#19 [debug] upstream https://208.67.220.2:443/dns-query exchanged ;www.arlingtoncemetery.mil.	IN	 A successfully in 368.826908ms
2023/10/27 16:03:50 539786#17 [debug] https://146.112.41.4:443/dns-query: response received over tcp: ok
2023/10/27 16:03:50 539786#17 [debug] upstream https://146.112.41.4:443/dns-query exchanged ;www.arlingtoncemetery.mil.	IN	 A successfully in 2.852382136s
2023/10/27 16:03:50 539786#18 [debug] https://208.67.222.2:443/dns-query: response received over tcp: ok
2023/10/27 16:03:50 539786#18 [debug] upstream https://208.67.222.2:443/dns-query exchanged ;www.arlingtoncemetery.mil.	IN	 A successfully in 3.448319279s

Quad9:

[root@ankino ~]# kdig @149.112.112.10 +https www.arlingtoncemetery.mil
;; TLS session (TLS1.3)-(ECDHE-X25519)-(ECDSA-SECP256R1-SHA256)-(AES-256-GCM)
;; HTTP session (HTTP/2-POST)-(149.112.112.10/dns-query)-(status: 200)
;; ->>HEADER<<- opcode: QUERY; status: SERVFAIL; id: 0
;; Flags: qr rd ra; QUERY: 1; ANSWER: 0; AUTHORITY: 0; ADDITIONAL: 1

;; EDNS PSEUDOSECTION:
;; Version: 0; flags: ; UDP size: 1232 B; ext-rcode: NOERROR

;; QUESTION SECTION:
;; www.arlingtoncemetery.mil.		IN	A

;; Received 54 B
;; Time 2023-10-27 16:20:29 CST
;; From 149.112.112.10@443(TCP) in 71.7 ms

OpenDNS:

[root@ankino ~]# kdig @208.67.222.2 +https www.arlingtoncemetery.mil
;; TLS session (TLS1.3)-(ECDHE-X25519)-(RSA-PSS-RSAE-SHA256)-(AES-256-GCM)
;; HTTP session (HTTP/2-POST)-(208.67.222.2/dns-query)-(status: 200)
;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 0
;; Flags: qr rd ra; QUERY: 1; ANSWER: 3; AUTHORITY: 0; ADDITIONAL: 1

;; EDNS PSEUDOSECTION:
;; Version: 0; flags: ; UDP size: 4096 B; ext-rcode: NOERROR
;; PADDING: 305 B

;; QUESTION SECTION:
;; www.arlingtoncemetery.mil.		IN	A

;; ANSWER SECTION:
www.arlingtoncemetery.mil.	1734	IN	CNAME	anc-web-prd-01-webelb-390236456.us-east-1.elb.amazonaws.com.
anc-web-prd-01-webelb-390236456.us-east-1.elb.amazonaws.com. 60	IN	A	54.144.81.3
anc-web-prd-01-webelb-390236456.us-east-1.elb.amazonaws.com. 60	IN	A	3.232.164.143

;; Received 468 B
;; Time 2023-10-27 16:21:49 CST
;; From 208.67.222.2@443(TCP) in 62.4 ms