bad upstream server in swarm mode?

[dugravot6]

Hi all,
In swarm mode (only one node for the moment), i create a stack of 3 services :

• nginx-proxy
• whoami
• acme-letsencrypt (not important)

When i try to reach the whoami container through nginx, i get an error like :

nginx_nginx-proxy.1.i4lo6342kqgk@minipc1    | nginx.1     | 2023/02/12 10:43:43 [error] 74#74: *5 upstream timed out (110: Operation timed out) while connecting to upstream, client: 10.0.0.2, server: whoami.local, request: "GET / HTTP/1.1", upstream: "http://10.0.0.254:8000/", host: "whoami.local"
nginx_nginx-proxy.1.i4lo6342kqgk@minipc1    | nginx.1     | whoami.local 10.0.0.2 - - [12/Feb/2023:10:43:43 +0000] "GET / HTTP/1.1" 504 167 "-" "curl/7.74.0" "10.0.0.254:8000"

When i try to reach the whoami container on his publish port (8001), it's ok :

curl -H "Host: whoami.local" 127.0.0.1:8001
I'm b62b642227f6

The problem is that nginx try to reach container via ingress network, not on the dedicated overlay (attachable) network (my-network) i setup before. We can see that it refers to the wrong network (ingress) in nginx configuration file :

upstream whoami.local {
    # Container: nginx_whoami.1.n4m2w81avbf4r0nzq9dw9ksnn
    #     networks: 
	#       ingress (reachable)
    #       my-network (ignored; reachable but redundant) 
    #     IP address: 10.0.0.254
    server 10.0.0.254:8000;
}

Here is my stack definition :

version: '3.4'

services:
  nginx-proxy:
    image: jwilder/nginx-proxy
    networks:
      - my-network
    ports:
      - "80:80"
        #      - "443:443"
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro
      - /home/DOCKER/NGINX/data/certs:/etc/nginx/certs
      - /home/DOCKER/NGINX/data/vhost.d:/etc/nginx/vhost.d
      - /home/DOCKER/NGINX/data/html:/usr/share/nginx/html
        #      - /home/DOCKER/NGINX/data/nginx.tmpl:/app/nginx.tmpl:ro
    deploy:
      mode: replicated
      replicas: 1
    labels:
      com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy: "true"

  whoami:
    image: jwilder/whoami
    networks:
      - my-network
    ports:
      - "8001:8000"
    depends_on:
      - nginx-proxy
    environment:
      - "VIRTUAL_HOST=whoami.local"
      - "LETSENCRYPT_HOST=whoami.local"
      - "LETSENCRYPT_EMAIL=stephane@example.com"

  acme-companion:
    image: nginxproxy/acme-companion
    networks:
      - my-network
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /home/DOCKER/NGINX/data/certs:/etc/nginx/certs
      - /home/DOCKER/NGINX/data/vhost.d:/etc/nginx/vhost.d
      - /home/DOCKER/NGINX/data/html:/usr/share/nginx/html
      - /home/DOCKER/ACME-COMPANION/data/acme/acme.sh:/etc/acme.sh
    depends_on:
      - nginx-proxy
    deploy:
      mode: replicated
      replicas: 1
    environment:
      - "DEFAULT_EMAIL=stephane@example.com"

networks:
  my-network:
    external: true

I don't know why nginx cannot join whoami container on the IP 10.0.0.254, and why i can't use my-network. Did you know what i have to do to nginx proxy works in swarm mode ?
Thank you.

Docker versions : 20.10.5+dfsg1
jwilder/nginx : latest (1.2.1)

[rhansen]

    ports:
      - "8001:8000"

What happens if you change ports: to expose: on your whoami service (but not your nginx-proxy service), like this:

    expose:
      - "8000"

[dugravot6]

@rhansen, it works :

# curl -H "Host: whoami.local" 127.0.0.1                                                                                                                                                               
I'm 11886f81f7d5

I have to understand why (probably the fact that : "ports will be accessible by other services connected to the same network"
I have also to verify if it's ok with other container.
Thank you @rhansen, you help me ;-)

[rhansen]

ports causes the specified ports to be published to the outside world. Your original config tells Docker to allow external users to access your container directly (not through nginx-proxy) via http://whoami.local:8001/, which is probably not what you want.

expose is only informative; it does nothing except add a bit of extra metadata to the container. nginx-proxy uses this metadata to figure out which port to use when connecting to your container. You can instead set the VIRTUAL_PORT environment variable and it would work the same.

To make the ports setting work, Docker swarm mode automatically attaches the container to a special ingress overlay mesh network. This network is used to route incoming connections from outside the swarm (and arriving at any swarm node) to a swarm node running one of the service's containers. It is not meant for internal container-to-container communication (such as requests from nginx-proxy to your application container).

I think your original config failed because:

• nginx-proxy does not support swarm mode (see Docker Swarm Support or Multiple Node Support #97), so it treats ingress like any other network.
• Docker does not permit internal container-to-container traffic on the ingress network. (I might be wrong about this; it might be due to a bug in the way nginx-proxy discovers ports when they are published via ports.)

[dakota-marshall]

If you use image: jwilder/nginx-proxy:1.1.0 it should work. While swarm isnt supported, we have been using it with swarm for awhile, and it broke for us this month. Using 1.1.0 works for us for the time being

[dakota-marshall]

It seems like in the most recent versions, dockergen seems to think that the container is reachable on the ingress network, when it isnt. 1.1.0 recognizes that it cannot be reached on the ingress network

[rhansen]

@dakota-marshall PR #2179 will restore the previous behavior.

I should note, however, that your service might be set up improperly. IIUC, Swarm tasks only attach to the ingress network if there is a port published to the outside world. If the port is published, clients could bypass nginx and access the application container directly (assuming there are no firewall rules blocking such requests).

Maybe your container is listening on multiple ports: one port is meant to be accessed via the nginx reverse proxy and the rest are intended for direct client access. In that case the container must be attached to the ingress network, hence PR #2179.

[dakota-marshall]

Hey @rhansen, thanks for the response! Ill keep an eye on that PR

You are correct, we are using the proxy for web traffic, but we also need to expose a different port directly through docker (Its gitlab, so we need SSH) So our container is getting both the ingress network and our proxy network.

[buchdag]

#2179 has been merged, I'll release a new version with the fix as soon as possible.

[dugravot6]

Thank you all !
Your help is really appreciated. Of course i have a firewall to block unwanted trafic. I will try today without "ports".

[dugravot6]

Finally, all my container didn't specify "ports" nor "expose" in docker-compose (stack - except nginx with ports 80 and 443). They did not have addresses in ingress network (except nginx even in 1.2.1 version it works), they all have an address in private overlay network of course (even nginx of course). And it's ok for all. It's pretty perfect for me.
Thank you again @rhansen and @dakota-marshall for your contribution.