Unable to filter traffic being forwarded from docker bridge interfaces

[gordonsyme]

Output of docker version:

Client:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 21:47:50 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 21:47:50 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 12
Server Version: 1.11.2
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 122
 Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge null host
Kernel Version: 3.13.0-86-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.676 GiB
Name: <redacted>
ID: <redacted>
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

Steps to reproduce the issue:

1. Set up iptables rules to block traffic from docker bridges to an external network

$ sudo iptables -L FORWARD -nv
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER-ISOLATION  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0

$ sudo /sbin/iptables --insert FORWARD 1 --in-interface docker+ --destination 169.254.0.0/16 --jump DROP
$ sudo /sbin/iptables --insert FORWARD 2 --in-interface br-+ --destination 169.254.0.0/16 --jump DROP

$ sudo iptables -L FORWARD -nv
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       all  --  docker+ *       0.0.0.0/0            169.254.0.0/16
    0     0 DROP       all  --  br-+   *       0.0.0.0/0            169.254.0.0/16
    0     0 DOCKER-ISOLATION  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0

2. Create a new bridge network

docker network create --driver=bridge test-network

3. Check if a container using this bridge can access the blocked network

$ docker run --rm --net=test-network curly curl http://169.254.169.254/latest/ 2>/dev/null
dynamic
meta-data

4. Observe that docker has created new rules in the FORWARD chain, before the rules restricting access to the external network, that allow any traffic originating from the docker bridge to be forwarded.

$ sudo iptables -L FORWARD -nv
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
31286   51M DOCKER-ISOLATION  all  --  *      *       0.0.0.0/0            0.0.0.0/0
   20  2138 DOCKER     all  --  *      br-f1a256c6d342  0.0.0.0/0            0.0.0.0/0
   20  2138 ACCEPT     all  --  *      br-f1a256c6d342  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
   20  1408 ACCEPT     all  --  br-f1a256c6d342 !br-f1a256c6d342  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  br-f1a256c6d342 br-f1a256c6d342  0.0.0.0/0            0.0.0.0/0
    5   300 DROP       all  --  docker+ *       0.0.0.0/0            169.254.0.0/16
    0     0 DROP       all  --  br-+   *       0.0.0.0/0            169.254.0.0/16
23412   50M DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0
23412   50M ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
 7829  426K ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0

The specific troublesome rule looks to be

ACCEPT     all  --  br-f1a256c6d342 !br-f1a256c6d342  0.0.0.0/0            0.0.0.0/0

which allows any traffic originating from the docker bridge to be forwarded anywhere.

Describe the results you expected:
I'd like to be able to enforce my own policies when it comes to filtering traffic. The ideal for me would be for docker to jump to a chain specifically intended for admins to add their own rules for traffic to/from docker bridge interfaces. Much like the jump to the DOCKER-ISOLATION chain, although my understanding is that DOCKER-ISOLATION is a managed chain that I shouldn't manipulate.

[cpuguy83]

This is pretty much a dup of #4737
You can certainly add rules wherever you like in the chains, but definitely do it in your own chain and insert your chains before docker's.

Note, I've created #23022 to help resolve these iptables issues.

[gordonsyme]

You can certainly add rules wherever you like in the chains, but definitely do it in your own chain and insert your chains before docker's.

Hmmm, I can't add jumps to my own chains in the FORWARD chain because any time I docker network create a new network, docker inserts rules at the beginning of that chain, bypassing mine.

Are you saying it's kosher for me to add a jump to a chain from DOCKER-ISOLATION? If so, is it the case that DOCKER-ISOLATION will only contain DROP rules and that the RETURN is guaranteed to always be the last item in DOCKER-ISOLATION, regardless of what I add?

Thanks

[cpuguy83]

@gordonsyme Oh I see, didn't realize libnetwork was prepending. I'm not sure why it would be pre-pending.

ping @mrjana

[mrjana]

ping @aboch

[aboch]

@gordonsyme
Please refer to https://github.com/docker/docker/blob/v1.12.0-rc2/docs/userguide/networking/default_network/container-communication.md

The forward chain you would use to insert user defined firewall rules is still the DOCKER one.
Bridge driver only appends rules in there when containers expose ports. But be aware that at daemon boot, the chain is wiped out.

$ sudo iptables -t filter -S DOCKER
-N DOCKER
-A DOCKER -d 172.18.0.3/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 90 -j ACCEPT

[gordonsyme]

@aboch AFAICS the jump to the DOCKER chain is only when the out interface is the docker bridge interface, there's no jump to that chain for traffic originating from the bridge interface and bound for the outside world.

I guess I could block the response traffic from the external network but that'd be an odd way to manage things.

[aboch]

@gordonsyme Ah right you are insterested in filtering traffic coming from the bridge, sorry.

There was a reason why the rules at network creation are pre-pended in the FORWARD chain, which I can't recall at the moment. Need to look into it if the need still holds true.

Maybe we should also think of adding a jump to DOCKER chain for traffic coming from the bridges, so that that chain can be used for this kind of user policies in both directions.

[gordonsyme]

Maybe we should also think of adding a jump to DOCKER chain for traffic coming from the bridges, so that that chain can be used for this kind of user policies in both directions.

This would be ideal

[GameScripting]

I also have the issue, that I cant filter on dockers traffic, because creating networks always prepends instead of appending.

Having a chain to put my own (DROP) rules without docker whiping them is a must.

[thaJeztah]

• closing in favour of Docker Network bypasses Firewall, no option to disable #22054 and [RFC] IPVS port mapper #45532