Installation breaks Ubuntu's systemd-resolved

[NavinF]

Versions

Pi-hole version is v5.11.4 (Latest: v5.11.4)
AdminLTE version is v5.13 (Latest: v5.13)
FTL version is v5.16.1 (Latest: v5.16.1)

Platform

• OS and version: Ubuntu 20.04 LTS
• Platform: x86-64

Expected behavior

pi-hole should never mess with the host system's DNS client

Actual behavior / bug

pi-hole screws up the host system's DNS client

Steps to reproduce

1. curl -sSL https://install.pi-hole.net | bash
2. ping google.com

This is likely related to systemd/systemd#14700

Running sudo systemctl status systemd-resolved prints DNSStubListener= is disabled, but /etc/resolv.conf is a symlink to /run/systemd/resolve/stub-resolv.conf which expects DNSStubListener= to be enabled.

I see the install log line [✓] Disabling systemd-resolved DNSStubListener and restarting systemd-resolved which is exactly the wrong thing to do regardless of the systemd bug. It leaves the server broken until I undo the change to /etc/systemd/resolved.conf.

IMO the ubuntu should not be listed as a supported OS if a default install on an LTS version breaks the system.

[yubiuser]

Relevant code:

pi-hole/automated install/basic-install.sh

Lines 1657 to 1678 in aefbe1f

	disable_resolved_stublistener() {
	printf " %b Testing if systemd-resolved is enabled\\n" "${INFO}"
	# Check if Systemd-resolved's DNSStubListener is enabled and active on port 53
	if check_service_active "systemd-resolved"; then
	# Check if DNSStubListener is enabled
	printf " %b %b Testing if systemd-resolved DNSStub-Listener is active" "${OVER}" "${INFO}"
	if ( grep -E '#?DNSStubListener=yes' /etc/systemd/resolved.conf &> /dev/null ); then
	# Disable the DNSStubListener to unbind it from port 53
	# Note that this breaks dns functionality on host until dnsmasq/ftl are up and running
	printf "%b %b Disabling systemd-resolved DNSStubListener" "${OVER}" "${TICK}"
	# Make a backup of the original /etc/systemd/resolved.conf
	# (This will need to be restored on uninstallation)
	sed -r -i.orig 's/#?DNSStubListener=yes/DNSStubListener=no/g' /etc/systemd/resolved.conf
	printf " and restarting systemd-resolved\\n"
	systemctl reload-or-restart systemd-resolved
	else
	printf "%b %b Systemd-resolved does not need to be restarted\\n" "${OVER}" "${INFO}"
	fi
	else
	printf "%b %b Systemd-resolved is not enabled\\n" "${OVER}" "${INFO}"
	fi
	}

[dschaper]

I see the install log line [✓] Disabling systemd-resolved DNSStubListener and restarting systemd-resolved which is exactly the wrong thing to do regardless of the systemd bug.

Pi-hole won't start if systemd-resolved is running.

[dschaper]

IMO the ubuntu should not be listed as a supported OS if a default install on an LTS version breaks the system.

Considering the number of Ubuntu installs that are working perfectly fine, I think it's a well supported OS for Pi-hole.

Now, if you'd like to see why your system is different then we're happy to take a look.

[NavinF]

Pi-hole won't start if systemd-resolved is running.

I don't follow. Is this because they both run on port 53? I set DNSStubListener=yes and have both of them running at the same time. systemd-resolved is running on 127.0.0.53:53 whereas pi-hole is on 127.0.0.1:53

$ sudo ss -tlnp|grep :53
LISTEN   0        32      127.0.0.1:53      0.0.0.0:*         users:(("pihole-FTL",pid=2077949,fd=17))
LISTEN   0        32      <LAN IP>:53      0.0.0.0:*         users:(("pihole-FTL",pid=2077949,fd=15))
LISTEN   0        4096     127.0.0.53%lo:53      0.0.0.0:*         users:(("systemd-resolve",pid=2059762,fd=13))
LISTEN   0        32      [::1]:53      [::]:*         users:(("pihole-FTL",pid=2077949,fd=43))

ping google.com uses systemd-resolved as expected whereas dig google.com @127.0.0.1 uses pi-hole. I think this should be the default config for pi-hole installations since it doesn't require messing with the rest of the system. (Well, pi-hole doesn't even need to bind to 127.0.0.1 but it's useful for debugging.)

Considering the number of Ubuntu installs that are working perfectly fine, I think it's a well supported OS for Pi-hole.
Now, if you'd like to see why your system is different then we're happy to take a look.

That's fair. I'm not sure what's special about my system and I'm not sure where to start looking. The only network related packages I've installed are docker and wireguard. I'm pretty sure it's just the systemd bug I linked that's screwed me over.

[outranker]

I am also encountering the same issue and I wonder what other package or app is using the port

[PromoFaux]

This, from the Readme on our Docker image, may be of some use:

https://github.com/pi-hole/docker-pi-hole#installing-on-ubuntu

[dschaper]

I don't follow. Is this because they both run on port 53?

Yes.

ubuntu@ip-172-26-3-153:~$ sudo ss -tlpn
State     Recv-Q    Send-Q       Local Address:Port       Peer Address:Port    Process
LISTEN    0         4096         127.0.0.53%lo:53              0.0.0.0:*        users:(("systemd-resolve",pid=403,fd=13))

ubuntu@ip-172-26-3-153:~$ sudo ./pihole-FTL-linux-x86_64 -f
 ...

dnsmasq: failed to create listening socket for port 53: Address already in use
[2022-08-04 00:16:47.338 1941M] FATAL ERROR in dnsmasq core: failed to create listening socket for port 53: Address already in use

Where there is a problem is that the installer is doing half the job. I'm not sure if something changed recently with systemd-resolved but now we need to make sure that we change the /etc/resolv.conf symlink to point to the non-stub config.

[jensbodal]

This, from the Readme on our Docker image, may be of some use:

https://github.com/pi-hole/docker-pi-hole#installing-on-ubuntu

link is now https://github.com/pi-hole/docker-pi-hole#installing-on-ubuntu-or-fedora

[debarkamondal]

I am new to open source. This is what my thinking pattern is currently, so please correct me if I am wrong.

If I am not getting it wrong the resolved.conf file is symlinked to some other file that is auto generated. So to do this I think you should break the symlink or delete and recreate the file with some sort or temp dns. Run the script. Go to the resolved.conf file again and set it to the dns you want.

[Slashic]

I've been scratching my head around this:

For people that wants to use systemd-resolved and its DNS over TLS capacity for instance, there's no way around using it, this is especially useful if you're using an authoritative DNS server/forwarder like BIND9 that does not handle DNS over TLS and setup your DNS forwarding this way:
Client -> PiHole -> Bind9 (optional) -> systemd-resolved -> DNS provider with TLS encryption

The problem is that dnsmasq by default tries to bind on 0.0.0.0 (everything):
If we change dnsmasq to port 54 for instance to avoid clashing we can clearly see this is the case:

# ss -tlnp|grep :53
LISTEN 0      4096      127.0.0.54:53         0.0.0.0:*    users:(("systemd-resolve",pid=2131,fd=16))
LISTEN 0      4096   127.0.0.53%lo:53         0.0.0.0:*    users:(("systemd-resolve",pid=2131,fd=14))
# ss -tlnp|grep :54
LISTEN 0      32           0.0.0.0:54         0.0.0.0:*    users:(("pihole-FTL",pid=3114,fd=5))
LISTEN 0      32              [::]:54            [::]:*    users:(("pihole-FTL",pid=3114,fd=7))

To avoid this, create a separate conf file for dnsmasq: /etc/dnsmasq.d/02-yourconf.conf

Add:

bind-interfaces

Save it and reload Pi-Hole service.

Dnsmasq will correctly bind the interfaces available and not 0.0.0.0 nor 127.0.0.53:

LISTEN 0      4096                              127.0.0.54:53         0.0.0.0:*    users:(("systemd-resolve",pid=2131,fd=16))
LISTEN 0      32                               192.168.1.9:53         0.0.0.0:*    users:(("pihole-FTL",pid=4377,fd=7))
LISTEN 0      32                                 127.0.0.1:53         0.0.0.0:*    users:(("pihole-FTL",pid=4377,fd=9))
LISTEN 0      4096                           127.0.0.53%lo:53         0.0.0.0:*    users:(("systemd-resolve",pid=2131,fd=14))

Hope that helps.

On a side note: I think this is a better approach than forcefully setting DNSStubListener=no that reverts systemd-resolved to an unknown behavior regarding the /etc/systemd/resolved.conf configuration without user warning, potentially loosing DNS encryption and corrupting /etc/resolv.conf stubbed file if /etc/systemd/resolved.conf contains other directives such as DNS servers for instance (it can creates duplicate entries upon /etc/resolv.conf refresh)