docker swarm mode: ports on 127.0.0.1 are exposed to 0.0.0.0

[fer2d2]

Description

In docker swarm mode, binding a port to 127.0.0.1 results with the port being open on 0.0.0.0 also. This could be a severe security issue and should be explained in the documentation.

Steps to reproduce the issue:

1. Create a service, for example MongoDB, in your docker-compose.swarm.yml file, and publish the port 27017 to localhost:

  mongodb:
    image: mongo:3.2
    volumes:
      - ./persistent-data/mongodb:/data
      - ./persistent-data/mongodb/db:/data/db
    networks:
      data:
        aliases:
          - mongo.docker
    logging:
      driver: syslog
      options:
        syslog-address: "udp://10.129.26.80:5514"
        tag: "docker[mongodb]"
    ports:
      - "127.0.0.1:27017:27017"
    deploy:
      placement:
        constraints: [node.labels.purpose == main-data]

2. Deploy your swarm
3. Check if the port is open from outside your swarm with netcat

Describe the results you received:

nc -vz PUBLIC_NODE_IP 27017
found 0 associations
found 1 connections:
[...]
Connection to PUBLIC_NODE_IP port 27017 [tcp/*] succeeded!

Describe the results you expected:
The port being only available on 127.0.0.1, at least in the swarm nodes running this service.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Docker version 17.03.1-ce, build c6d412e

Output of docker info:

docker info for swarm manager:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 17.03.1-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 3
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: active
 NodeID: pk7ulemi0z0chgtsg0azfrjz5
 Is Manager: true
 ClusterID: 27etomlyjvtmygrm6rcdgr2ni
 Managers: 1
 Nodes: 6
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 10.129.26.165
 Manager Addresses:
  10.129.26.165:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-64-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 992.4 MiB
Name: <HIDDEN>
ID: IMOK:QIR7:WU5Y:WTPP:EPRQ:F77G:ULGE:WOG4:O7S7:6AFE:V7QG:2XEK
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: <HIDDEN>
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):
Tested on Digital Ocean's droplets.

[thaJeztah]

Yes this should output an error; services (by default) "publish" using the "ingress" network, and do not support specifying an IP-address, as it's not possible to predict which node they end up on (thus not known what IP-addresses are available - although 127.0.0.1 could be possible). This issue is tracking that feature #26696 (and this "epic" tracks other options not (yet) supported by services #25303)

The bug here is that docker should produce an error, instead of silently ignoring the option; reproducible using this minimal docker-compose file;

version: "3.2"
services:
  mongodb:
    image: nginx:alpine
    ports:
      - "127.0.0.1:27017:80"

[thaJeztah]

ping @dnephin @vdemeester

[vdemeester]

@fer2d2 On swarm mode, if you publish something (ports for stack deploy), it is published on the ingress network, and thus it is public. There is a few ways to get around, but putting kind/bug on that because we should at least warn people about that when doing a stack deploy with ports that have this notation (i.e. host:port:port).

To work around this, there is a few ways:

• first, you should publish mongo ports only if you want it to be public, otherwise, it is available through the name discovery bundle in docker (another container/service on the same network will be able to reach it through mongo dns name).
• If you want to publish it in the host and not in ingress (so not swarm public, just on the host it is running, same way as without swarm mode), you need to use ports expanded syntax.

    ports:
      - mode: host
        target: 80
        published: 9005

It will do the same as docker run -p 80:9005 … so it will bind it to 0.0.0.0, but on limited to the host.

But as @thaJeztah said, "The bug here is that docker should produce an error, instead of silently ignoring the option" 👼

/cc @mavenugo @aboch to see if there would be a way to actually being able to bind it to a specific ip ? (really tricky to achieve because node's ip will be different so..)

[fer2d2]

@vdemeester Could I specify localhost as the host target using this notation?

    ports:
      - mode: host
        target: 127.0.0.1:80
        published: 9005

As it is an extended format for ports configuration, it should work properly.

Thanks in advance

[bsimpson53]

It seems that both target and published are enforced as integer type in the long syntax

[fer2d2]

I think this is not the desired behaviour if you are connecting to some services via SSH tunnels. For example, if you want to have your MySQL or MongoDB server on 127.0.0.1 and connect via SSH Tunnel, with Docker Swarm you must expose the database port on 0.0.0.0 or create a custom database container with SSH running inside (and both options are very insecure).

There are many database clients using SSH tunnels, like SQL Workbench or Robomongo that can't be used due to this limitation (specific interface binding).

[alvaromr]

We have the same problem in our company as @fer2d2, trying to connect Mongobooster with a docker swarm via ssh tunnel. The only solution we found was opening 27017 port and protecting the database with user and password.

[soar]

Any news?

[smessmer]

+1

[megglos]

+1

[rjmaxwell]

Another use case for allowing ip_address:port pair for long form port mapping is for anycast addresses or any other addresses that may be associated to loopback. These would be similar to a 127.0.0.1 address in that they are only visible on the loopback network. A service restricted to nodes with this property may wish to expose a port only on an anycast address in order to avoid port collisions while avoiding iptables rules for port translation.

[bhartm3]

Can it possibly be an option when you specify:

placement:
        constraints:
          - node.id ==

Cheers

[ivibe]

+1

[awgneo]

+1

[pbullian]

+1

[helldweller]

for myself I solved this problem so:

iptables -I DOCKER-USER -i eth0 -j DROP
iptables -I DOCKER-USER -m state --state RELATED,ESTABLISHED -j ACCEPT

The docker does not touch these rules. just adds your own
-A DOCKER-USER -j RETURN
As a result, although the port listens on 0.0.0.0 but is not accessible from the external interface eth0

[tad-lispy]

I like your workaround @maxisme, but how do you manage the authorized_keys ownership? On OS X it works for me (the mount belongs to root) but on a production Linux machine I'm getting:

Authentication refused: bad ownership or modes for file /root/.ssh/authorized_keys
Connection closed by authenticating user root 85.145.195.174 port 60535 [preauth]

The volume belongs to UID of the host user, which is not root and SSHD refuses to work with it. A workaround on top of your workaround 😬 is to use configs, like this:

services:
  sshd:
    image: [...]/sshd:${version}
    configs:
      # FIXME: It would be much better to use a bind volume for this, as it
      # would always be in sync with the host configuration. So revoking a key
      # in the host machine would automatically revoke it in the container. But
      # I can't figure out how to give the volume right ownership. It keeps UID
      # from the host which doesn't align with the container user.
      - source: authorized_keys
        target: /root/.ssh/authorized_keys
        mode: 0600

configs:
  authorized_keys:
    file: ~/.ssh/authorized_keys

[marrem]

I understand that due to the fact that you don't know what host a container will be deployed to you cannot tell the service to bind to a specific host ip address.

However often hosts have e.g. north and south bound interfaces. You might want the swarm ports to bind only to the northbound interfaces on all swarm hosts.

If the interface names of all the interfaces you want a service to bind to are the same (e.g. eth0), it might be an idea to offer an option to specify an interfacename to bind swarm ports to (in service ports section).

    nginx:
      image: nvbeta/swarm_nginx
      networks:
        - demonet1
      ports:
        - "eth0:8088:80"

When eth0 is not available on a swarm node the specified port won't be bound to any interface.

[DrPyser]

@tad-lispy You should be able to change uid and gid of the container user to be the same as the volume owner on the host.
The linuxserver image supports this by setting environment variables (see https://hub.docker.com/r/linuxserver/openssh-server, User / Group Identifiers),

[swift1337]

Found great article for Ubuntu server users
https://p1ngouin.com/posts/how-to-manage-iptables-rules-with-ufw-and-docker

[MicahZoltu]

This should really be prioritized as a security issue. The docs currently say that 127.0.01:80:80 works (https://docs.docker.com/compose/compose-file/compose-file-v3/#short-syntax-1) and when you do that you open your container up to the world instead of localhost like the docs say.

I'm not expecting a hotfix to make this behave the way people want, but it definitely SHOULD NOT open a security hole when the user follows the recommendations in the documentation. It should hard error or at least be correct in the docs.

[sky93]

Any updates on this?

[erwinschaap]

I hope this is high urgent and will soon be fixed, because this will potentially lead to major data breaches.

[ambs]

Got hacked because of this. Ok, my fault, but still...

[wereii]

There is another solution, at least for bare metal/servers with virtualization available.

Run docker in virtual machine

Pros

•  ports:
     - mode: host

  Will now actually open ports only on/to your local host, yay!
• Clean ip a on your host
• Clean iptables-save on your host
• Nothing will ever bypass your carefully crafted firewall rules ever again
• Even better separation of concerns, Enhanced Security ™️
• Better resource control

Cons

• Yes you have to ssh twice (easy?)
• Getting Swarm up requires bridging your VLAN interfaces into the VMs (still rather easy?)

[jspraul]

Heads up: #22054 (comment) (2021-11-05)

docker containers in the 172.16.0.0/12 ip range are definitely reachable from external hosts. All it requires is a custom route on the attacker's machine that directs traffic bound for 172.16.0.0/12 through the victim's machine.

Notice that the common advice of binding the published port to 127.0.0.1 does not protect the container from external access.

...

You don't even have to --publish.

[john8329]

After a bunch of hours researching how to prevent this, I found in the documentation that there's the DOCKER-USER iptables chain for adding rules, that might help blocking ports. Although I used an external firewall for simplicity because iptables is cumbersome.
iptables -I DOCKER-USER -i eno1 -j DROP

edit: somebody mentioned already here. Whoops.

[Fedcomp]

I hit the same issue when wanting to access traefik dashboard on different port bound to 127.0.0.1:8080. This way i could use ssh but it listens on 0.0.0.0 instead ... and this issue is from 2017 ...

[matveynator]

Assignees
No one assigned

2017 and now is 2024 that is sick ...

[esemeniuc]

I solved this problem without socat, which is an extra hop between userspace and kernel

1. Expose the port

    ports:
      - target: 26379 # note: public (internet facing) access is blocked via iptables, see below.
        published: 26379
        mode: host

2. Use iptables to block the port from the internet:

# allow only traffic from loopback (localhost).
sudo iptables -t mangle -I PREROUTING ! -i lo -p tcp --dport 26379 -j DROP

Explanation:
Docker has a PREROUTING rule on the nat table (see here for Linux packet flow). We must insert our rule before Docker's rule, so mangle is our only option.

[mghahari]

Publishing in host mode and blocking with iptables are both unscalable.
This is a use case:
We deploy fluentbit on our cluster for log processing with docker log driver. Deploying it on host mode causes that for each new node to our cluster we should define a new service with placement configuration for that node. This is not scalable.
If we deploy it on ingress mode we should add this firewall rule to each node we add and is not scalable.

[shcorya]

Here's a scalable solution using UNIX sockets.

Compose file:

version: '3.8'

services:

  mk-socket-dir:
    image: alpine
    command: mkdir -p /run/test
    volumes:
      - /run:/run
    deploy:
      mode: global-job

  socket-in:
    image: alpine/socat
    command: "-dd TCP-L:8081,fork,bind=localhost UNIX:/run/test/test.sock"
    volumes:
      - /run/test:/run/test
    networks:
      - public
    deploy:
      mode: global

  socket-out:
    image: alpine/socat
    command: "-dd UNIX-L:/run/test/test.sock,fork TCP:whoami:80"
    volumes:
      - /run/test:/run/test
    networks:
      - internal
    deploy:  
      mode: global

  # listens on port 80
  whoami:
    image: traefik/whoami
    hostname: whoami
    networks:
      - internal

networks:
  internal:
    driver: overlay
  public:
    name: host
    external: true

Test:

curl -s http://localhost:8081

[docwhat]

Here's a scalable solution using UNIX sockets.

Since it shares data over a volume, it won't work if there are multiple hosts in your swarm.

[mghahari]

Here's a scalable solution using UNIX sockets.

Since it shares data over a volume, it won't work if there are multiple hosts in your swarm.

The service deploy mode is global so this volume will be available on each host. @docwhat

[docwhat]

I'm sorry I overlooked that detail. I don't use Swarm anymore.

It's a neat solution to route a localhost port, even if you have to run two containers per host per port.