[Bug] "Replacing resolv.conf" fails without error #76

ExceptionGit · 2021-06-09T15:37:44Z

When /etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf no replacement and dns leaking via systemd-resolved nameserver 127.0.0.53.

orjail/usr/sbin/orjail

Lines 227 to 229 in a0b9e6c

    
           print G " * Replacing resolv.conf..." 
        
           mount --bind -o users "$RESOLVEFILE" /etc/resolv.conf || 
        
             die "Failed to mount /etc/resolv.conf."

The text was updated successfully, but these errors were encountered:

phantomcraft · 2021-06-10T00:11:51Z

This bug happens because a mount point with --bind option cannot be a symbolic link, it needs to be a file or a folder.

I had this problem before.

Perhaps uninstalling the package resolvconf in Debian/Ubuntu should solve this issue.

ExceptionGit · 2021-06-10T08:23:08Z

I don't want to remove this symbolic link: systemd-resolved.

How about redirect via iptables?

phantomcraft · 2021-06-10T23:59:44Z

There is no way to forward DNS at port 53 to a uplayer IP.

Let's suppose a veth pair inside a netns sandbox:

veth --> 10.0.0.1
peer --> 10.0.0.2

iptables cannot forward DNS queries to 10.0.0.1, the DNS port must be at 10.0.0.2 (locally) or 127.0.0.1. For achieving this you should run a UDP port forwarder inside the sandbox to forward 10.0.0.1:53 to 127.0.0.1:53 (or 127.0.0.1:9053). An so add this iptables rule:

iptables -t nat -A PREROUTING -p udp --dport 53 -j REDIRECT --to-ports 9053

There is a UDP port forwarder called udpxd, it can do the job:

https://github.com/TLINDEN/udpxd

Run the udpxd inside the orjail

udpxd --listen 127.0.0.1:9053 --to 10.200.135.2:53 --daemon

Add a iptables rule:

sudo iptables -t nat -A PREROUTING -p udp --dport 53 -j REDIRECT --to-ports 9053

ExceptionGit · 2021-06-12T12:34:17Z

I think it does not work: 1 2 3.

~~I always get curl: (6) Could not resolve host: 3g2upl4pq6kufc4m.onion and no DNS REDIRECT/DNAT~~

Ok, i found 2 issues when systemd-resolved used,

Symbolic link enabled /etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf, run $ orjail -v -s
- in verbose mode: Replacing resolv.conf...
- $ cat /etc/resolv.conf: nameserver 10.xxx.xxx.xxx - OK
- wait x Min to update systemd-resolved
- $ cat /etc/resolv.conf: nameserver 127.0.0.53 - BAD
- no errors in verbose mode
Curl "? uses systemd-resolved" and ignores /etc/resolv.conf, run $ orjail -v -s
- $ curl 3g2upl4pq6kufc4m.onion: curl: (6) Could not resolve host: 3g2upl4pq6kufc4m.onion
- wait x Min, nameserver 127.0.0.53
- $ dig github.com: connection timed out; no servers could be reached
- $ curl 3g2upl4pq6kufc4m.onion: curl: (6) Could not resolve host: 3g2upl4pq6kufc4m.onion
- $ curl https://github.com: <!DOCTYPE html>..</html>
- dns leaks and no errors in verbose mode

phantomcraft · 2021-06-12T23:51:49Z

If you forward the 53 port from veth to loopback and apply the iptables rule for forwarding, I'm sure it'll work.

phantomcraft · 2021-06-13T06:57:21Z

I tested the UDP port forwarding inside orjail and it works.

Start orjail:

sudo orjail -s

Find the address of orjail veth:

ip address show in-orjail
59: in-orjail@if58: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 3a:fc:ce:0b:ac:23 brd ff:ff:ff:ff:ff:ff link-netns orjail
    inet 10.200.47.2/30 scope global in-orjail
       valid_lft forever preferred_lft forever
    inet6 fe80::38fc:ceff:fe0b:ac23/64 scope link 
       valid_lft forever preferred_lft forever

Run udpxd inside orjail pointing to the address of veth

udpxd --listen 127.0.0.1:9053 --to 10.200.47.2:53 --daemon

Add iptables rule:

sudo iptables -t nat -A PREROUTING -p udp --dport 53 -j REDIRECT --to-ports 9053

Test it:

user@localhost:~$ curl 3g2upl4pq6kufc4m.onion
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>

ExceptionGit · 2021-06-13T10:03:03Z

This only works if you haven't installed systemd-resolved is a systemd service that provides network name resolution to local applications via a D-Bus interface . And yes, i think curl (and maybe other apps) uses systemd-resolved via D-Bus.

My tests inside the orjail:

curl: $ rm /etc/resolv.conf OR $ echo 'nameserver 127.0.0.53' > /etc/resolv.conf OR $ echo 'nameserver 10.200.XXX.2' > /etc/resolv.conf
- $ curl github.com - OK
- $ curl 3g2upl4pq6kufc4m.onion - BAD
- $ curl domain.local - resolve OK, but Recv failure: Connection reset by peer
- $ resolvectl query github.com - OK
- $ resolvectl query 3g2upl4pq6kufc4m.onion - BAD
- $ resolvectl query domain.local - OK
dig does not use D-Bus
- $ rm /etc/resolv.conf - ALL BAD
- $ echo 'nameserver 127.0.0.53' > /etc/resolv.conf - ALL BAD
- $ echo 'nameserver 10.200.XXX.2' > /etc/resolv.conf - ALL OK, .local BAD

I found a simple solution (inside the orjail:):

$ sudo orjail -s
$ sudo sysctl -w net.ipv4.conf.all.route_localnet=1 - enable forwarding for loopback interface, to prevent a error dig: internal_send: 127.0.0.53#53: Invalid argument
sudo iptables -t nat -A OUTPUT -o lo -p udp --dport 53 -j DNAT --to-destination 10.200.XXX.2:5354 - dns forwarding
sudo iptables -t nat -A POSTROUTING -o out-orjail -j SNAT --to 10.200.XXX.1 - change source ip

It does not prevent dns leaks via systemd-resolved (curl, etc..), but works without udpxd.

How about add this behavior if /etc/resolv.conf is symbolic link?

phantomcraft · 2021-06-16T04:21:36Z

There is no need for iptables rules, as by default the resolv.conf in network namespaces are placed at: /etc/<netns_name/resolv.conf

I'm contributor but @lesion should commit if he want.

Instead of creating a resolv.conf and bind to /etc/resolv.conf, a folder /etc/netns should be created and a sub-folder /etc/netns/orjail too, in which a resolv.conf should be placed. This is the correct way.

ExceptionGit · 2021-06-16T16:50:07Z

$ sudo orjail -v -s
 * ...
 * Creating a orjail namespace...
 * ...

$ sudo mkdir -p /etc/netns/orjail/
$ sudo cp /tmp/resolve..... /etc/netns/orjail/resolv.conf

$ dig github.com
;; connection timed out; no servers could be reached

$ curl 3g2upl4pq6kufc4m.onion
curl: (6) Could not resolve host: 3g2upl4pq6kufc4m.onion

phantomcraft · 2021-06-17T00:58:38Z

It works here:

root@localhost:/home/user# rm /etc/resolv.conf
root@localhost:/home/user# echo "nameserver 1.1.1.1" > /dev/shm/resolv.conf
root@localhost:/home/user# ln -s /dev/shm/resolv.conf /etc/resolv.conf
root@localhost:/home/user# mkdir -p /etc/netns/orjail
root@localhost:/home/user# mv /tmp/resolveGbUN6e /etc/netns/orjail/
root@localhost:/home/user# orjail -s

user@localhost:~$ host g.co
g.co has address 172.217.169.142
Host g.co not found: 3(NXDOMAIN)
Host g.co not found: 4(NOTIMP)

The only way is to use iptables rules in your case. In orjail, when an error happens in the /etc/resolv.conf mount, a trigger that runs iptables should be added, @lesion should implement this.

phantomcraft · 2021-06-20T07:55:13Z

O modified orjail to deal with /etc/resolv.conf if it's a symbolic link:

orjail.zip

Test to see if it works.

I added this lines:

  if [ "$(ls -l /etc/resolv.conf | cut -d " " -f 10)" = "->" ]; then
    ip netns exec "$NAME" iptables -t nat -A OUTPUT -p udp --dport \
      53 -j DNAT --to-destination $IPHOST:5354
  fi

ExceptionGit · 2021-06-20T12:35:19Z

Without:

sysctl -w net.ipv4.conf.all.route_localnet=1                            # enable forwarding for loopback interface
                                                                        # https://serverfault.com/questions/211536/
iptables -t nat -A POSTROUTING -o out-orjail -j SNAT --to 10.200.XXX.1  # change source ip

it will not work if nameserver 127.0.0.53 in the /etc/resolv.conf.

Now I use this:

cat ~/bin/orjail-dns.sh

#!/bin/bash

IP_HOST=$(ip addr show type veth | grep -Po 'inet \K(\.|\d)+')
IP_SERVER=$(ip addr show type veth | grep -Po 'inet \K(\d+\.\d+\.\d+\.)')'2'

echo 'fix dns:'
sudo sysctl -w net.ipv4.conf.all.route_localnet=1 1>/dev/null
sudo iptables -t nat -A OUTPUT -o lo -p udp --dport 53 -j DNAT --to-destination $IP_SERVER:5354
sudo iptables -t nat -A POSTROUTING -o out-orjail -j SNAT --to $IP_HOST

/bin/bash

sudo orjail -v ~/bin/orjail-dns.sh

ExceptionGit · 2021-09-15T11:08:46Z

Any news or updates? In firejail it's works without iptables rules. And about the dns leak netblue30/firejail#2869 (comment)

lesion · 2021-09-15T16:12:17Z

I think the issue here is with nsswitch.conf that uses resolve instead of dns:

nss-resolve is a plug-in module for the GNU Name Service Switch (NSS) functionality of the GNU C Library (glibc) enabling it to resolve hostnames via the systemd-resolved(8) local network name resolution service. It replaces the nss-dns plug-in module that traditionally resolves hostnames via DNS.

my proposal is to solve this issue by mounting our nsswitch.conf version (as we do with resolv.conf) to force glibc/gethostbyname to use our resolv.conf avoiding any other resolution's type (systemd-resolve included).
it could be something like:

            hosts: dns files
            passwd:files
            shadow:files
            group:files
            hosts:dns files
            bootparams:files
            ethers:files
            netmasks:files
            networks:files
            protocols:files
            rpc:files
            services:files
            automount:files
            aliases:files

I also do not understand the symlink issue you have with /etc/resolv.conf:

debian# ls -la /etc/resolv.conf 
lrwxrwxrwx 1 root root 29 26 nov  2020 /etc/resolv.conf -> ../run/resolvconf/resolv.conf

debian# echo test > test1
debian# mount --bind test1 /etc/resolv.conf
debian# cat /etc/resolv.conf               
test

lesion · 2021-09-15T20:57:38Z

@ExceptionGit could you test this solution in feat/nsswitch branch ?
thanks

ExceptionGit · 2021-09-16T04:34:26Z

Test "feat/nsswitch branch" - curl work, no dns leak.

Symlink issue sudo orjail -v /bin/bash:

# orjail
$ ls /etc/resolv.conf
/etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf
$ cat /etc/resolv.conf
nameserver 10.200.XXX.2

# host
$ ls /etc/resolv.conf
/etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf
$ cat /etc/resolv.conf
nameserver 127.0.0.53

# wait XX time or restart network on host
$ systemctl restart systemd-networkd.service

# orjail
$ cat /etc/resolv.conf
nameserver 127.0.0.53     <--- THIS without errors in verbose mode (/etc/nsswitch.conf is OK)

lesion · 2021-09-16T09:24:35Z

I'm not able to reproduce this 😢 , also this sounds really strange to me (I cannot understand how this could be possible), but I pushed another commit to modify the logic and use /etc/netns to "overwrite" resolv.conf and nsswitch.conf instead of mount-binding them as @phantomcraft suggested. I don't think this does change something as ip netns is doing the same thing but can you try it? thanks

lesion · 2021-09-16T09:51:59Z

interesting thread: slingamn/namespaced-openvpn#7

-> https://github.com/slingamn/namespaced-openvpn#dns-hardening

ExceptionGit · 2021-09-16T23:04:57Z

Similar result after systemctl restart systemd-networkd.service

How about firejail implementation?

$ firejail --net=XXXX --dns=XXX.XXX.XXX.XXX -- bash
$ cat /etc/resolv.conf
nameserver XXX.XXX.XXX.XXX
$ ls /etc/resolv.conf
/etc/resolv.conf            <-- no symlink

$ systemctl restart systemd-networkd.service

$ cat /etc/resolv.conf
nameserver XXX.XXX.XXX.XXX

lesion · 2021-09-17T10:11:43Z

using /etc/netns/orjail/resolv.conf does not solve the issue: as pointed out here https://unix.stackexchange.com/questions/418304/why-do-linux-bind-mounts-disappear-if-the-mount-points-inode-changes:

One case where this is relevant in practice is ip-netns(8); ip netns exec works by bind mounting /etc/netns/${NAMESPACE}/resolv.conf on top of /etc/resolv.conf. If the inode of /etc/resolv.conf is altered by resolvconf(8) or systemd-resolved, the updated /etc/resolv.conf will be visible to the process running inside the namespace.

What about mount-binding the whole /etc and replacing relevant files (resolv.conf/nsswitch.conf)?
This is pratically how firejail does this

This is done in 8bdfe75, I'm confident this is the way to go.

ExceptionGit · 2021-09-18T00:09:30Z

I think this is the best way, everything works now.

Maybe completely remove $USERHOME/.orjail/ (root:root) in

orjail/usr/sbin/orjail

Lines 154 to 155 in 8bdfe75

    
           print G " * Remove temporary /etc dir ($ETC_DIR)..." 
        
           eno rm -fr "$ETC_DIR"

lesion · 2021-09-20T08:29:24Z

Merged in master.
We cannot remove $USERHOME/.orjail because we could have multiple namespaces

ExceptionGit · 2021-09-20T11:41:03Z

Remove if this is the last namespace? It is very strange to get the directory in home ~/.orjail/ where the owner is root:root.

What happens if the ETC_DIR has not been removed previously? (hard reset, power off)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug] "Replacing resolv.conf" fails without error #76

[Bug] "Replacing resolv.conf" fails without error #76

ExceptionGit commented Jun 9, 2021

phantomcraft commented Jun 10, 2021

ExceptionGit commented Jun 10, 2021

phantomcraft commented Jun 10, 2021 •

edited

ExceptionGit commented Jun 12, 2021 •

edited

phantomcraft commented Jun 12, 2021

phantomcraft commented Jun 13, 2021

ExceptionGit commented Jun 13, 2021

phantomcraft commented Jun 16, 2021 •

edited

ExceptionGit commented Jun 16, 2021

phantomcraft commented Jun 17, 2021

phantomcraft commented Jun 20, 2021

ExceptionGit commented Jun 20, 2021

ExceptionGit commented Sep 15, 2021

lesion commented Sep 15, 2021

lesion commented Sep 15, 2021

ExceptionGit commented Sep 16, 2021

lesion commented Sep 16, 2021

lesion commented Sep 16, 2021 •

edited

ExceptionGit commented Sep 16, 2021 •

edited

lesion commented Sep 17, 2021

ExceptionGit commented Sep 18, 2021 •

edited

lesion commented Sep 20, 2021

ExceptionGit commented Sep 20, 2021

[Bug] "Replacing resolv.conf" fails without error #76

[Bug] "Replacing resolv.conf" fails without error #76

Comments

ExceptionGit commented Jun 9, 2021

phantomcraft commented Jun 10, 2021

ExceptionGit commented Jun 10, 2021

phantomcraft commented Jun 10, 2021 • edited

ExceptionGit commented Jun 12, 2021 • edited

phantomcraft commented Jun 12, 2021

phantomcraft commented Jun 13, 2021

ExceptionGit commented Jun 13, 2021

phantomcraft commented Jun 16, 2021 • edited

ExceptionGit commented Jun 16, 2021

phantomcraft commented Jun 17, 2021

phantomcraft commented Jun 20, 2021

ExceptionGit commented Jun 20, 2021

ExceptionGit commented Sep 15, 2021

lesion commented Sep 15, 2021

lesion commented Sep 15, 2021

ExceptionGit commented Sep 16, 2021

lesion commented Sep 16, 2021

lesion commented Sep 16, 2021 • edited

ExceptionGit commented Sep 16, 2021 • edited

lesion commented Sep 17, 2021

ExceptionGit commented Sep 18, 2021 • edited

lesion commented Sep 20, 2021

ExceptionGit commented Sep 20, 2021

phantomcraft commented Jun 10, 2021 •

edited

ExceptionGit commented Jun 12, 2021 •

edited

phantomcraft commented Jun 16, 2021 •

edited

lesion commented Sep 16, 2021 •

edited

ExceptionGit commented Sep 16, 2021 •

edited

ExceptionGit commented Sep 18, 2021 •

edited