How to manually add site conf's for proxy_pass (127.0.0.1) in docker rootless

[Sieboldianus]

I have decided to migrate my Mastodon docker to rootless.

I have created to users that run in separate namespaces.

The first stack is Mastodon (5 services):

services:
  web:
    ports:
       - "127.0.0.1:3000:3000"
    environment:
      # VIRTUAL_HOST: "mastodon.mytld.com"
      # VIRTUAL_PATH: "/"
      # VIRTUAL_PORT: 3000
      # LETSENCRYPT_HOST: "mastodon.mytld.com"
      # LETSENCRYPT_EMAIL: "mail@mytld.com"
  streaming:
    ports:
      - "127.0.0.1:4000:4000"
    environment:
      # VIRTUAL_HOST: "mastodon.mytld.com"
      # VIRTUAL_PATH: "/api/v1/streaming"
      # VIRTUAL_PORT: 4000
[...]

The nginx-stack:

version: "3.7"

services:
    reverse-proxy:
        image: jwilder/nginx-proxy:latest
        container_name: reverse-proxy
        ports:
            - "0.0.0.0:80:80"
            - "0.0.0.0:443:443"
        volumes:
            - /srv/nginx/data/certs:/etc/nginx/certs:ro
            - /srv/nginx/data/config/dhparam:/etc/nginx/dhparam
            - /srv/nginx/data/config/html:/usr/share/nginx/html
            - /srv/nginx/data/config/vhost.d:/etc/nginx/vhost.d
            - /srv/nginx/data/config/conf.d:/etc/nginx/conf.d
            - /srv/nginx/data/logs:/var/log/nginx
            - ${XDG_RUNTIME_DIR}/docker.sock:/tmp/docker.sock:ro
        environment:
            - SSL_POLICY=Mozilla-Modern
            - ENABLE_IPV6=true
            - TRUST_DOWNSTREAM_PROXY=false
        restart: always
        networks:
            - net

    letsencrypt:
        image: nginxproxy/acme-companion
        container_name: letsencrypt-helper
        volumes_from:
            - reverse-proxy
        volumes:
            - /srv/nginx/data/certs:/etc/nginx/certs:rw
            - /srv/nginx/data/acme:/etc/acme.sh
            - ${XDG_RUNTIME_DIR}/docker.sock:/var/run/docker.sock:ro
        environment:
            DEFAULT_EMAIL: "mail@mytld.com"
        restart: always
        networks:
            - net

networks:
  net:
    external: true

I wanted to separate nginx and my service stack because I will be adding more docker services and I don't want all to run in the same userspace. This means I will have to proxy through port bindings on localhost. The services as stated above run fine, no errors, server is reachable on localhost via curl.

However, in this case all commented environment parameters are useless (VIRTUAL_HOST, VIRTUAL_PATH, VIRTUAL_PORT, LETSENCRYPT_EMAIL, LETSENCRYPT_HOST).

What would be the simplest way to reach the same effect by manually modifying/adding these to the nginx stack?

I tried creating a file at:

~/data/config/vhost.d/mastodon.mytld.com_location_override

with the following content (according to the Readme):

location / {
    proxy_pass http://127.0.0.1:3000;
}

But it does not seem to have any effect.

I don't want to create my own nginx.conf, but use what is provided as a default by nginx-proxy, as if I were to use VIRTUAL_HOST, VIRTUAL_PATH (etc.) in the same docker network.

I don't see any other users having this problem, which is strange since docker rootless gained traction.

[buchdag]

Hi.

Could you elaborate on what you mean by

However, in this case all commented environment parameters are useless (VIRTUAL_HOST, VIRTUAL_PATH, VIRTUAL_PORT, LETSENCRYPT_EMAIL, LETSENCRYPT_HOST).

Why are those environment variables commented out in your Mastodon compose files ?

On which network are those containers, the default one created with the compose file (won't work) or net ?

[Sieboldianus]

These values are used to inform nginx-proxy about the ports, paths etc.

I cannot use these variables if nginx-proxy and mastodon are on completely separated namespaces/networks that can only communicate via port binding to (127.0.0.1:3000).

I reverted back to using the system nginx as a reverse proxy to forward data to the rootless Mastodon docker via 127.0.0.1:3000 and 127.0.0.1:4000. However, this required writing a manual nginx.conf. This works, but it would still be intereresting to see how this could be done with nginx-proxy automatically.

[buchdag]

I might be missing something here but why don't you connect the Matsodon containers to the net network ? Having the proxy and the proxyed containers sharing at least one network is one of the basic requirements for usage of nginx-proxy.

[Sieboldianus]

Yes, this was basically my question - I had it set up like this before docker rootless, but this concept is not compatible with rootless separating individual services into individual networks. It means I cannot use nginx-proxy with Docker rootless unless I use only one docker rootless and bind the docker rootless socket into the nginx-proxy (and have a single docker network connecting all my containers).

[buchdag]

I don't know much about Docker rootless since I'm not actively using it but I don't immediately see the correlation between being rootless and not being able to put containers in the Docker networks you want.

Could you point me to the documentation on this ?

[Sieboldianus]

Yes, here: https://docs.docker.com/engine/security/rootless/

And you are correct, again. These are the bullet points of my thoughts:

• Docker in rootless can equally function as rootfull docker. E.g., everything in one (rootless) docker, including nginx-proxy on the same docker network and bind-mounting the (rootless) dockert socket into the nginx-container
• This means, however, that I cannot separate services, which is the main point of rootless: Different services into different containers with individual rootless docker sockets
• ideally, I want a single nginx reverse proxy, either isolated in its own rootless docker system, or installed on the host
• then packets are routed to individual separated services through localhost binded ports
• this means, nginx is not on the same network as the services; it is a consequence of my system setup, not a limitation of the rootless docker

I think there really isn't much to say more - it is my own dilemma. I could use nginx-proxy with docker rootless, but in a way that I don't like (or from which I am trying to get away with by using rootless docker in the first place).

After writing my initial question above, I already set up Mastodon this way now.

• The Mastodon Service stack is in a rootless docker network
• it has two external ports listening on 127.0.0.1:3000 and 127.0.0.1:4000
• nginx on the host does the SSL termination and forwards packets to :3000 and :4000 on localhost, to Mastodon

Should Mastodon get hacked, the compromised data would be limited to the Mastodon rootless docker/rootless user, not the whole system.

[buchdag]

Should Mastodon get hacked, the compromised data would be limited to the Mastodon rootless docker/rootless user, not the whole system.

That's my thought too, I'm no security expert but your reverse proxy is by nature already exposed to the outside, hacking a proxied service shouldn't give an attacker any more leverage on the reverse proxy service, there are no backdoor or service ports opened only to internal networks.

That's a different story if you use the local network access feature though.

By the way as acme-companion don't need to publish any privileged port it should be able to run in rootless too (and it should theoretically not need to be in the same Docker network as the other containers either because it's only communicating over network with the outside).