Feature HA Proxy protocol support

[hitech95]

What type of PR?

Enhancement

What does this PR do?

This PR implements PROXY protocol for email endpoint on front. This allow to have the front container behind another email proxy. With PROXY protocol we can forward the Origin-IP/Client-IP wich allows admin to properly apply throttling rules also behind a proxy. Enabling the PROXY_PROTOCOL all the mail endpoints expect it.

Custom endpoints have also been added to provide auth only endpoints. this allow to have SSL termination in the proxy and STARTLS available on port 25 without having to forward TLS traffic with the proxy.
Those endpoins are not expected to be mapped to the host so they are not been added to the Dockerfile EXPOSE list.

Related issue(s)

• Properly use traefik or any proxy for SSL termination: Mailu behind Traefik 2.2 #1422
• Allow to have front behind a mail proxy (like another nginx instance) : closes PROXY Protocol support #1472

[mergify]

Thanks for submitting this pull request.
Bors-ng will now build test images. When it succeeds, we will continue to review and test your PR.

bors try

Note: if this build fails, read this.

[bors]

try

Build failed:

• CI-Done

[nextgens]

Why do you create new endpoints instead of disabling STARTTLS in nginx when EMAIL_PROXYED is true?

Shouldn't it imply TLS = false ?

[hitech95]

Why do you create new endpoints instead of disabling STARTTLS in nginx when EMAIL_PROXYED is true?

Shouldn't it imply TLS = false ?

Some proxies dont play nicely with STARTTLS. That port set allow to have the front container to handle StartTLS and the proxy to handle SSL termination.

What I want to have only SSL but allow STATTLS on port 25? I cannot SSL terminate to a STARTTLS endpoint.

[nextgens]

I don't understand what you mean. Can you clarify the usecase for this (why do you need a proxy in the first place, why does it need to do the SSL termination, ... )?

Some proxies dont play nicely with STARTTLS. That port set allow to have the front container to handle StartTLS and the proxy to handle SSL termination.

That's the worst of both worlds: you end up with SSL termination in two different places -> you need to deploy valid certificates in both.

[hitech95]

I don't understand what you mean. Can you clarify the usecase for this (why do you need a proxy in the first place, why does it need to do the SSL termination, ... )?

Some proxies dont play nicely with STARTTLS. That port set allow to have the front container to handle StartTLS and the proxy to handle SSL termination.

That's the worst of both worlds: you end up with SSL termination in two different places -> you need to deploy valid certificates in both.

What if you want to have SSL provided by front but non encrypted local/lan endpoints for internal services?
The two endpoints for the webmail that follows this concept have special throttling rules.

This allow to have SSL, StartTLS, and not encrypted endpoints available and also support for PROXY protocol.

[nextgens]

I don't understand what you mean. Can you clarify the usecase for this (why do you need a proxy in the first place, why does it need to do the SSL termination, ... )?

Some proxies dont play nicely with STARTTLS. That port set allow to have the front container to handle StartTLS and the proxy to handle SSL termination.

That's the worst of both worlds: you end up with SSL termination in two different places -> you need to deploy valid certificates in both.

What if you want to have SSL provided by front but non encrypted local/lan endpoints for internal services? The two endpoints for the webmail that follows this concept have special throttling rules.

You can use the proxy to export a port without SSL (where the client would connect without SSL to the proxy and the proxy would establish an SSL connection to Mailu).

In Mailu we specifically discourage people to run a mailserver without TLS (except for testing, see https://mailu.io/master/compose/setup.html#tls-flavor).

This allow to have SSL, StartTLS, and not encrypted endpoints available and also support for PROXY protocol.

I think that allowing for the proxy protocol to be supported is an enhancement of what we already do (by allowing HTTP reverse proxies); I do not think that exposing non encrypted endpoints brings anything useful to the table (and I do think that it's a bad idea, going against what we are attempting to do elsewhere: ensure that clients connect in a secure way).

[hitech95]

You can use the proxy to export a port without SSL (where the client would connect without SSL to the proxy and the proxy would establish an SSL connection to Mailu).

I think that only nginx would allow somehting like that. Most proxies are not able to do so. This reduce versability.

In Mailu we specifically discourage people to run a mailserver without TLS (except for testing, see https://mailu.io/master/compose/setup.html#tls-flavor).

You can always be enabled and it is independent if you need the not secure port set.

I think that allowing for the proxy protocol to be supported is an enhancement of what we already do (by allowing HTTP reverse proxies); I do not think that exposing non encrypted endpoints brings anything useful to the table (and I do think that it's a bad idea, going against what we are attempting to do elsewhere: ensure that clients connect in a secure way).

Right now ther are 2 endpoints that are not encryped and not exposed: the two used by the webmail.
The portset I've added is the same concept but they are a full port set without encryption, with authentication, and standard login throtteling rules.

Let's look to some examples:

• Proxy can SSL terminate but cannot do STARTTLS. (like HA proxy and Traefik). You must SSL terminate (TLS_FLAVOR!=notls ) in in the front controller. If you have a service that require access to mail in the same mailu network your forced to go via SSL and the proxy. You cannot go straight to the front without SSL. Does having SSL in the internal hopefully "trusted" network be necessary?

• Proxy can SSL Terminate and do STARTTLS. (like nginx). You specify TLS_FLAVOR=notls and you're good to go with original portset. Internal services can connect directly to front.

If you think that this is not necessary I can drop that section.

[hitech95]

@hitech95 , my understanding is that this enhancement would allow to use a load-balancers on typical managed k8s clusters (as you say that it would close #1472).

So, one of the main use-cases for this functionality would be to "enable mailu deployment on managed k8s with load-balancers", which feels like an important feature.

I'm using in my docker stack with Traefik in front of front. Traefik is configured with HA proxy protocol so I can forward the source IP address. Nginx and Ha proxy can be used as a proxy too! So k8s Ha might be one of the applications.

[nextgens]

@abebeos if what you are looking for is images with the patch, you can get them from https://hub.docker.com/r/mailuci/

https://hub.docker.com/r/mailuci/nginx/ for pr2300

We're definitely not merging code that hasn't been tested and to me it's very far from clear that this additional functionality does not break anything.

[nextgens]

bors try

[bors]

try

Build succeeded:

• CI-Done

[fischerscode]

(Only talking about PROXY_PROTOCOL)

This change is mandatory when running Mailu behind a reverse proxy. This scenario might occur when using a single floating IP for incoming mail and web traffic in a Kubernetes environment. (My use case. Details can be found here.)

Please consider merging those changes. I think it would be a nice addition when having clustered setups in mind.

I cant speak about EMAIL_PROXYED, since I don't really know when it would be used.

[fischerscode]

to me it's very far from clear that this additional functionality does not break anything.

Since PROXY_PROTOCOL and EMAIL_PROXYED are not true in any existing environment (since they are introduced here), nothing should be added to nginx.conf and therefore nothing should break.

Edit: When it comes to testing it in an environment that uses the PROXY_PROTOCOL feature, I could help if you want me to do so.

[nextgens]

We've discussed this at the last dev-meeting (sorry this isn't in the minutes)... To reach a mergeable state we need:

• A clearly defined and documented usecase. It can be k8s/ingress or traefik to something else but we need to have a documented way of making use of the feature
• I've come to term with EMAIL_PROXYED but not the name of the configuration knob and not adding new ports; port 25 will always require STARTTLS support; if the proxy can't do it, it means that we will need certs in both places; What about PROXY_WITH_STARTTLS or PROXY_CAN_DO_STARTTLS and removing the new ports (make the existing ports do starttls or not depending on the new knob)? I can't see a usecase where both are useful in conjunction; If the proxy can do STARTTLS it can export its own ports not requiring encryption.
• REAL_IP_FROM needs to actually be enforced. It's one thing to get it wrong with HTTP... it's potentially opening up open-relays once we are talking about proxying mail ports.
• Why isn't PROXY_PROTOCOL not controlling what's done for HTTP too? If the proxy is compatible with it we shouldn't do HTTP based proxying IMHO.
• It needs to be tested and confirmed working

[nextgens]

reformulating the above about EMAIL_PROXYED, I see two cases: we either have:

• a STARTTLS compatible proxy (nginx for instance), in which case we will get TLS_FLAVOUR=none and the certs will not be in front
• a non-compatible proxy, in which case we will get TLS_FLAVOUR=mail* and the certs will have to be both in front and in the proxy

The PR as currently proposed adds new ports in the case where we have a compatible proxy and I'm arguing that it's not useful since the compatible proxy can export its own "listeners" with or without SSL on its own.

[hitech95]

• A clearly defined and documented usecase. It can be k8s/ingress or traefik to something else but we need to have a documented way of making use of the feature

I can provide my use case. (see below)

• I've come to term with EMAIL_PROXYED but not the name of the configuration knob and not adding new ports; port 25 will always require STARTTLS support; if the proxy can't do it, it means that we will need certs in both places; What about PROXY_WITH_STARTTLS or PROXY_CAN_DO_STARTTLS and removing the new ports (make the existing ports do starttls or not depending on the new knob)? I can't see a usecase where both are useful in conjunction; If the proxy can do STARTTLS it can export its own ports not requiring encryption.

See my example below.

• REAL_IP_FROM needs to actually be enforced. It's one thing to get it wrong with HTTP... it's potentially opening up open-relays once we are talking about proxying mail ports.

Yes this is a must, I dont remember exactly but without it NGINX don't route the traffic (need testing).

• Why isn't PROXY_PROTOCOL not controlling what's done for HTTP too? If the proxy is compatible with it we shouldn't do HTTP based proxying IMHO.

The email TCP proxy and web http(s) proxy could be different this meas that both protocols should be different. One could use nginx for email end apache for http(s).

My docker stacks:
Traefik:

version: "3.3"

networks:
  public:
    external: true
    name: macvlan-public
  proxy:
    internal: true
    name: proxy
    ipam:
      driver: default
      config:
        - subnet: 172.77.77.0/24
  relay:
    external: false
    internal: true
    ipam:
      driver: default
      config:
        - subnet: 172.70.70.0/24
volumes:
  traefik-certs:
    driver_opts:
      type: "nfs"
      o: "addr=192.168.1.55,nfsvers=3,noatime,rsize=8192,wsize=8192,tcp,timeo=14"
      device: ":/export/docker-traefik-certs"

services:
  traefik:
    image: "traefik:v2.6"
    container_name: "Traefik"
    restart: always
    environment:
      - PUID=100
      - PGID=1000
      - TZ=Europe/Rome
    command:
      - "--log.level=INFO"
      - "--accesslog=true"
      # Docker configuration
      - "--providers.docker=true"
      - "--providers.docker.network=proxy"
      - "--providers.docker.exposedbydefault=false"
      # Configure entrypoint
      # HTTP(S)
      - "--entrypoints.api.address=:8080"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--entryPoints.web.forwardedHeaders.trustedIPs=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22"
      - "--entryPoints.websecure.forwardedHeaders.trustedIPs=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22"
      # EMAIL(S)
      - "--entrypoints.smtp.address=:25"
      - "--entrypoints.smtps.address=:465"
      - "--entrypoints.smtp-auth.address=:587"
      - "--entrypoints.imap.address=:143"
      - "--entrypoints.imaps.address=:993"
      - "--entrypoints.pop.address=:110"
      - "--entrypoints.pops.address=:995"
      # SSL configuration
      - "--certificatesresolvers.letsencrypt.acme.dnschallenge=true"
      - "--certificatesresolvers.letsencrypt.acme.dnschallenge.provider=cloudflare"
      - "--certificatesresolvers.letsencrypt.acme.email=mymail@myisp.com"
      - "--certificatesresolvers.letsencrypt.acme.storage=/certs/acme.json"
      # Global HTTP -> HTTPS
      - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
      - "--entrypoints.web.http.redirections.entryPoint.scheme=https"
      # Enable dashboard
      - "--api.dashboard=true"
    labels:
      # Global middlewares
      - "traefik.http.middlewares.api-whitelist.ipwhitelist.sourcerange=127.0.0.1/32, 192.168.1.0/24"
      # Traefik Dashboard
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.rule=Host(`traefik.mydomain.com`)"
      - "traefik.http.routers.traefik.entrypoints=api"
      - "traefik.http.routers.traefik.service=api@internal"
      # Fake Service port for the router
      - "traefik.http.services.traefik.loadbalancer.server.port=8888"
    ports:
      - 80:80
      - 443:443
    networks:
      public:        
      proxy:
        ipv4_address: 172.77.77.254
      relay:
        ipv4_address: 172.70.70.254
    volumes:
      - "traefik-certs:/certs"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

  certdumper:
    image: humenius/traefik-certs-dumper:1.5.0
    container_name: "Traefik-Dumper"
    restart: always
    environment:
      - PUID=100
      - PGID=1000
      - TZ=Europe/Rome
      - ACME_FILE_PATH=/traefik/acme.json
    networks:
      - proxy
    depends_on:
      - traefik
    volumes:
      - "traefik-certs:/traefik:ro"
      - "traefik-certs:/output"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

Mailu stack:

version: "2.2"

networks:
  mailu:
    external: false
    #internal: true
    name: mailu
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 172.69.69.0/24
          gateway: 172.69.69.1
  proxy:
    external: true
    name: proxy

volumes:
  mailu-certs:
    external: true
    name: proxy_traefik-certs
  mailu-redis:
  mailu-postgres:
  mailu-front:
  mailu-admin:
  mailu-dkim:
  mailu-imap:
  mailu-smtp:
  mailu-rspamd:
  mailu-webdav:
  mailu-clamav:
  mailu-fetchmail:
  mailu-webmail:

services:
  # External dependencies
  resolver:
    container_name: "mailu-unbound"
    image: mailu/unbound:1.9
    restart: always
    environment:
      - PUID=100
      - PGID=1000
      - TZ=Europe/Rome
      # UNBOUND
      - SUBNET=172.69.69.0/24
    networks:
      mailu:
        ipv4_address: 172.69.69.254

  redis:
    container_name: "mailu-redis"
    image: redis:alpine
    restart: always
    environment:
      - PUID=100
      - PGID=1000
      - TZ=Europe/Rome
    networks:
      - mailu
    volumes:
      - "mailu-redis:/data"
    depends_on:
      - resolver
    dns:
      - 172.69.69.254

  postgres:
    image: postgres:11
    container_name: "mailu-db"
    restart: always
    environment:
      - PUID=100
      - PGID=1000
      - TZ=Europe/Rome
      # Postgres
      - POSTGRES_USER=mailu
      - POSTGRES_PASSWORD=xyz
      - POSTGRES_DB=mailu
    networks:
      - mailu
    volumes:
      - "mailu-postgres:/var/lib/postgresql/data"
    depends_on:
      - resolver
    dns:
      - 172.69.69.254

  pgadmin:
    container_name: "mailu-pgadmin"
    image: dpage/pgadmin4
    restart: always
    environment:
      PGADMIN_DEFAULT_EMAIL: admin@admin.com
      PGADMIN_DEFAULT_PASSWORD: password
    networks:
      - mailu
    ports:
      - "5050:80"

  # Core services
  front:
    container_name: "mailu-front"
    # image: "mailu/nginx:1.9"
    image: "hitech95/mailu-nginx:1.9-feat-haproxy"
    restart: always
    environment:
      - PUID=100
      - PGID=1000
      - TZ=Europe/Rome
      # NGINX
      - HOSTNAMES=mail.mydomain.com
      - KUBERNETES_INGRESS=true
      - REAL_IP_HEADER=X-Forwarded-For
      - REAL_IP_FROM=172.77.77.254
      - EMAIL_PROXYED=true
      - PROXY_PROTOCOL=true
      - WEBROOT_REDIRECT=none
      - WEB_ADMIN=/admin
      - ADMIN=true
      - WEBMAIL=none
      - WEBDAV=none
      - MESSAGE_SIZE_LIMIT=500000000
      # START TLS
      - "TLS_FLAVOR=mail"
      - "TLS_CERT_FILENAME=mail.mydomain.com/cert.pem"
      - "TLS_KEYPAIR_FILENAME=mail.mydomain.com/key.pem"
    labels:
      # WEB UI
      - "traefik.enable=true"
      - "traefik.http.services.mailu-web.loadbalancer.server.port=80"
      - "traefik.http.services.mailu-web.loadbalancer.server.scheme=http"
      - "traefik.http.routers.mailu-web.rule=Host(`mail.mydomain.com`)"
      - "traefik.http.routers.mailu-web.entrypoints=web,websecure"
      - "traefik.http.routers.mailu-web.tls=true"
      - "traefik.http.routers.mailu-web.tls.certresolver=letsencrypt"
      - "traefik.http.routers.mailu-web.tls.domains[0].main=mail.mydomain.com"
      - "traefik.http.routers.mailu-web.tls.domains[0].sans=imaps.mydomain.com,imap.mydomain.com,pop3s.mydomain.com,pop3.mydomain.com,smtps.gremydomain.com,smtp.mydomain.com"
      - "traefik.http.routers.mailu-web.service=mailu-web"
      # EMAIL
      # SMTP Relay (START TLS)
      - "traefik.tcp.services.mailu-smtp.loadbalancer.server.port=25"
      - "traefik.tcp.services.mailu-smtp.loadbalancer.proxyProtocol=true"
      - "traefik.tcp.routers.mailu-smtp.rule=HostSNI(`mail.mydomain.com`, `smtp.mydomain.com`, `*`)"
      - "traefik.tcp.routers.mailu-smtp.entrypoints=smtp"
      - "traefik.tcp.routers.mailu-smtp.tls=false"
      - "traefik.tcp.routers.mailu-smtp.service=mailu-smtp"
      # SMTP Submission (START TLS)
      # - "traefik.tcp.services.mailu-submission.loadbalancer.server.port=587"
      # - "traefik.tcp.services.mailu-submission.loadbalancer.proxyProtocol=true"
      # - "traefik.tcp.routers.mailu-submission.rule=HostSNI(`mail.mydomain.com`, `smtp.mydomain.com`, `*`)"
      # - "traefik.tcp.routers.mailu-submission.entrypoints=smtp-auth"
      # - "traefik.tcp.routers.mailu-submission.tls=false"
      # - "traefik.tcp.routers.mailu-submission.service=mailu-submission"
      # SMTP Submission (SSL)
      - "traefik.tcp.services.mailu-smtps.loadbalancer.server.port=465"
      - "traefik.tcp.services.mailu-smtps.loadbalancer.proxyProtocol=true"
      - "traefik.tcp.routers.mailu-smtps.rule=HostSNI(`mail.mydomain.com`, `smtps.mydomain.com`)"
      - "traefik.tcp.routers.mailu-smtps.entrypoints=smtps"
      - "traefik.tcp.routers.mailu-smtps.tls.passthrough=true"
      - "traefik.tcp.routers.mailu-smtps.service=mailu-smtps"
      # IMAP (START TLS)
      # - "traefik.tcp.services.mailu-imap.loadbalancer.server.port=143"
      # - "traefik.tcp.services.mailu-imap.loadbalancer.proxyProtocol=true"
      # - "traefik.tcp.routers.mailu-imap.rule=HostSNI(`mail.mydomain.com`, `imap.mydomain.com`, `*`)"
      # - "traefik.tcp.routers.mailu-imap.entrypoints=imap"
      # - "traefik.tcp.routers.mailu-imap.passthrough=true"
      # - "traefik.tcp.routers.mailu-imap.service=mailu-imap"
      # IMAP (SSL)
      - "traefik.tcp.services.mailu-imaps.loadbalancer.server.port=993"
      - "traefik.tcp.services.mailu-imaps.loadbalancer.proxyProtocol=true"
      - "traefik.tcp.routers.mailu-imaps.rule=HostSNI(`mail.mydomain.com`, `imaps.mydomain.com`)"
      - "traefik.tcp.routers.mailu-imaps.entrypoints=imaps"
      - "traefik.tcp.routers.mailu-imaps.tls.passthrough=true"
      - "traefik.tcp.routers.mailu-imaps.service=mailu-imaps"
      # POP3 (START TLS)
      # - "traefik.tcp.services.mailu-pop3.loadbalancer.server.port=110"
      # - "traefik.tcp.services.mailu-pop3.loadbalancer.proxyProtocol=true"
      # - "traefik.tcp.routers.mailu-pop3.rule=HostSNI(`mail.mydomain.com`, `pop3.mydomain.com`, `*`)"
      # - "traefik.tcp.routers.mailu-pop3.entrypoints=pop"
      # - "traefik.tcp.routers.mailu-pop3.tls=false"
      # - "traefik.tcp.routers.mailu-pop3.service=mailu-pop3"
      # POP3 (SSL)
      # - "traefik.tcp.services.mailu-pop3s.loadbalancer.server.port=11110"
      # - "traefik.tcp.services.mailu-pop3s.loadbalancer.proxyProtocol=true"
      # - "traefik.tcp.routers.mailu-pop3s.rule=HostSNI(`mail.mydomain.com`, `pop3s.mydomain.com`)"
      # - "traefik.tcp.routers.mailu-pop3s.entrypoints=pops"
      # - "traefik.tcp.routers.mailu-pop3s.passthrough=true"
      # - "traefik.tcp.routers.mailu-pop3s.service=mailu-pop3s"
    networks:
      - mailu
      - proxy
    volumes:
      - "mailu-front:/overrides:ro"
      - "mailu-certs:/certs:ro"
    depends_on:
      - resolver
      - admin
    dns:
      - 172.69.69.254

  admin:
    container_name: "mailu-admin"
    image: mailu/admin:1.9
    restart: always
    environment:
      - PUID=100
      - PGID=1000
      - TZ=Europe/Rome
      # App
      - DISABLE_STATISTICS=true
      - SECRET_KEY=xyz
      - RECIPIENT_DELIMITER=+
      # Network
      - SUBNET=172.69.69.0/24
      - DOMAIN=mydomain.com
      - HOSTNAMES=mail.mydomain.com,servername.mydomain.com
      - POSTMASTER=admin
      # Auth
      - AUTH_RATELIMIT_IP=60/hour
      - AUTH_RATELIMIT_USER=100/day
      # DB Connection
      - DB_FLAVOR=postgresql
      - DB_HOST=postgres
      - DB_USER=mailu
      - DB_PW=mypassword
      - DB_NAME=mailu
    networks:
      - mailu
    volumes:
      - "mailu-admin:/data"
      - "mailu-dkim:/dkim"
    depends_on:
      - redis
      - postgres
    dns:
      - 172.69.69.254

  imap:
    container_name: "mailu-imap"
    image: mailu/dovecot:1.9
    restart: always
    environment:
      - PUID=100
      - PGID=1000
      - TZ=Europe/Rome
      # DOVECOT
      - DOMAIN=mydomain.com
      - HOSTNAMES=mail.mydomain,servername.mydomain.com
      - SUBNET=172.69.69.0/24
      - POSTMASTER=admin
      - RECIPIENT_DELIMITER=+
    networks:
      - mailu
    volumes:
      - "mailu-imap:/mail"
    depends_on:
      - resolver
      - front
    dns:
      - 172.69.69.254

  smtp:
    container_name: "mailu-smtp"
    image: mailu/postfix:1.9
    restart: always
    environment:
      - PUID=100
      - PGID=1000
      - TZ=Europe/Rome
      # POSTFIX
      - DOMAIN=mydomain.com
      - HOSTNAMES=mail.mydomain.com,servername.mydomain.com
      - SUBNET=172.69.69.0/24
      - RELAYNETS=172.70.70.0/24
      - RECIPIENT_DELIMITER=+
      - MESSAGE_SIZE_LIMIT=500000000
      - MESSAGE_RATELIMIT=200/day
    networks:
      - mailu
    volumes:
      - "mailu-smtp:/queue"
    depends_on:
      - resolver
      - front
    dns:
      - 172.69.69.254

  antispam:
    container_name: "mailu-antispam"
    image: mailu/rspamd:1.9
    restart: always
    environment:
      - PUID=100
      - PGID=1000
      - TZ=Europe/Rome
      # RSPAMD
      - SUBNET=172.69.69.0/24
      - RELAYNETS=172.70.70.0/24
      - ANTIVIRUS=clamav
    networks:
      - mailu
    volumes:
      - "mailu-rspamd:/var/lib/rspamd"
      - "mailu-dkim:/dkim"
    depends_on:
      - resolver
      - front
    dns:
      - 172.69.69.254

An example of my stack that use local relay:

version: '3'
networks:
  web:
    external: true
    name: proxy
  relay:
    external: true
    name: relay
  vaultwarden:
    external: false
    name: vaultwarden
    
volumes:
  bitwarden:
    external: true
    name: bitwarden
  postgres:
    external: true
    name: bitwarden_db
    
services:
  postgres:
    image: postgres:11
    container_name: vaultwarden_pgsql
    restart: always
    environment:
      - PUID=100
      - PGID=1000
      - TZ=Europe/Rome
      - POSTGRES_USER=vaultwarden
      - POSTGRES_PASSWORD=xyz
      - POSTGRES_DB=bitwarden
    networks:
      - vaultwarden
    volumes:
      - postgres:/var/lib/postgresql/data
      
  vaultwarden:
    image: vaultwarden/server:latest
    container_name: vaultwarden
    restart: always
    environment:
      - PUID=100
      - PGID=1000
      - TZ=Europe/Rome
      # Vaultwarden
      - WEBSOCKET_ENABLED=true  # Enable WebSocket notifications.
      - DATABASE_URL=postgresql://vaultwarden:xyz@postgres/bitwarden
      - ADMIN_TOKEN=xyz
    networks:
      - web
      - relay
      - vaultwarden
    labels:
      - "traefik.enable=true"
      # Two services are running
      # See: https://github.com/SergioBenitez/Rocket/issues/90
      # And: https://github.com/dani-garcia/vaultwarden/wiki/Enabling-WebSocket-notifications
      - "traefik.http.services.vaultwardenUI.loadbalancer.server.port=80"
      - "traefik.http.services.vaultwardenSocket.loadbalancer.server.port=3012"      
      # Route for UI
      - "traefik.http.routers.vaultwardenUI.rule=Host(`bitwarden.mydomain.com`)"
      - "traefik.http.routers.vaultwardenUI.entrypoints=websecure"
      - "traefik.http.routers.vaultwardenUI.tls.certresolver=letsencrypt"
      - "traefik.http.routers.vaultwardenUI.service=vaultwardenUI"
      # Route for Websockets
      - "traefik.http.routers.vaultwardenSocket.rule=Host(`bitwarden.mydomain.com`) && Path(`/notifications/hub`)"
      - "traefik.http.routers.vaultwardenSocket.entrypoints=websecure"
      - "traefik.http.routers.vaultwardenSocket.tls.certresolver=letsencrypt"      
      - "traefik.http.routers.vaultwardenSocket.service=vaultwardenSocket"
    volumes:
      - bitwarden:/data
    depends_on:
      - postgres

Local relay settings are:
host: 172.70.70.254
port 25
no auth

[samos667]

• A clearly defined and documented usecase. It can be k8s/ingress or traefik to something else but we need to have a documented way of making use of the feature

I can provide my use case. (see below)

• I've come to term with EMAIL_PROXYED but not the name of the configuration knob and not adding new ports; port 25 will always require STARTTLS support; if the proxy can't do it, it means that we will need certs in both places; What about PROXY_WITH_STARTTLS or PROXY_CAN_DO_STARTTLS and removing the new ports (make the existing ports do starttls or not depending on the new knob)? I can't see a usecase where both are useful in conjunction; If the proxy can do STARTTLS it can export its own ports not requiring encryption.

See my example below.

• REAL_IP_FROM needs to actually be enforced. It's one thing to get it wrong with HTTP... it's potentially opening up open-relays once we are talking about proxying mail ports.

Yes this is a must, I dont remember exactly but without it NGINX don't route the traffic (need testing).

• Why isn't PROXY_PROTOCOL not controlling what's done for HTTP too? If the proxy is compatible with it we shouldn't do HTTP based proxying IMHO.

The email TCP proxy and web http(s) proxy could be different this meas that both protocols should be different. One could use nginx for email end apache for http(s).

My docker stacks: Traefik:

version: "3.3"

networks:
  public:
    external: true
    name: macvlan-public
  proxy:
    internal: true
    name: proxy
    ipam:
      driver: default
      config:
        - subnet: 172.77.77.0/24
  relay:
    external: false
    internal: true
    ipam:
      driver: default
      config:
        - subnet: 172.70.70.0/24
volumes:
  traefik-certs:
    driver_opts:
      type: "nfs"
      o: "addr=192.168.1.55,nfsvers=3,noatime,rsize=8192,wsize=8192,tcp,timeo=14"
      device: ":/export/docker-traefik-certs"

services:
  traefik:
    image: "traefik:v2.6"
    container_name: "Traefik"
    restart: always
    environment:
      - PUID=100
      - PGID=1000
      - TZ=Europe/Rome
    command:
      - "--log.level=INFO"
      - "--accesslog=true"
      # Docker configuration
      - "--providers.docker=true"
      - "--providers.docker.network=proxy"
      - "--providers.docker.exposedbydefault=false"
      # Configure entrypoint
      # HTTP(S)
      - "--entrypoints.api.address=:8080"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--entryPoints.web.forwardedHeaders.trustedIPs=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22"
      - "--entryPoints.websecure.forwardedHeaders.trustedIPs=173.245.48.0/20,103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,141.101.64.0/18,108.162.192.0/18,190.93.240.0/20,188.114.96.0/20,197.234.240.0/22,198.41.128.0/17,162.158.0.0/15,104.16.0.0/13,104.24.0.0/14,172.64.0.0/13,131.0.72.0/22"
      # EMAIL(S)
      - "--entrypoints.smtp.address=:25"
      - "--entrypoints.smtps.address=:465"
      - "--entrypoints.smtp-auth.address=:587"
      - "--entrypoints.imap.address=:143"
      - "--entrypoints.imaps.address=:993"
      - "--entrypoints.pop.address=:110"
      - "--entrypoints.pops.address=:995"
      # SSL configuration
      - "--certificatesresolvers.letsencrypt.acme.dnschallenge=true"
      - "--certificatesresolvers.letsencrypt.acme.dnschallenge.provider=cloudflare"
      - "--certificatesresolvers.letsencrypt.acme.email=mymail@myisp.com"
      - "--certificatesresolvers.letsencrypt.acme.storage=/certs/acme.json"
      # Global HTTP -> HTTPS
      - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
      - "--entrypoints.web.http.redirections.entryPoint.scheme=https"
      # Enable dashboard
      - "--api.dashboard=true"
    labels:
      # Global middlewares
      - "traefik.http.middlewares.api-whitelist.ipwhitelist.sourcerange=127.0.0.1/32, 192.168.1.0/24"
      # Traefik Dashboard
      - "traefik.enable=true"
      - "traefik.http.routers.traefik.rule=Host(`traefik.mydomain.com`)"
      - "traefik.http.routers.traefik.entrypoints=api"
      - "traefik.http.routers.traefik.service=api@internal"
      # Fake Service port for the router
      - "traefik.http.services.traefik.loadbalancer.server.port=8888"
    ports:
      - 80:80
      - 443:443
    networks:
      public:        
      proxy:
        ipv4_address: 172.77.77.254
      relay:
        ipv4_address: 172.70.70.254
    volumes:
      - "traefik-certs:/certs"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

  certdumper:
    image: humenius/traefik-certs-dumper:1.5.0
    container_name: "Traefik-Dumper"
    restart: always
    environment:
      - PUID=100
      - PGID=1000
      - TZ=Europe/Rome
      - ACME_FILE_PATH=/traefik/acme.json
    networks:
      - proxy
    depends_on:
      - traefik
    volumes:
      - "traefik-certs:/traefik:ro"
      - "traefik-certs:/output"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"

Mailu stack:

version: "2.2"

networks:
  mailu:
    external: false
    #internal: true
    name: mailu
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 172.69.69.0/24
          gateway: 172.69.69.1
  proxy:
    external: true
    name: proxy

volumes:
  mailu-certs:
    external: true
    name: proxy_traefik-certs
  mailu-redis:
  mailu-postgres:
  mailu-front:
  mailu-admin:
  mailu-dkim:
  mailu-imap:
  mailu-smtp:
  mailu-rspamd:
  mailu-webdav:
  mailu-clamav:
  mailu-fetchmail:
  mailu-webmail:

services:
  # External dependencies
  resolver:
    container_name: "mailu-unbound"
    image: mailu/unbound:1.9
    restart: always
    environment:
      - PUID=100
      - PGID=1000
      - TZ=Europe/Rome
      # UNBOUND
      - SUBNET=172.69.69.0/24
    networks:
      mailu:
        ipv4_address: 172.69.69.254

  redis:
    container_name: "mailu-redis"
    image: redis:alpine
    restart: always
    environment:
      - PUID=100
      - PGID=1000
      - TZ=Europe/Rome
    networks:
      - mailu
    volumes:
      - "mailu-redis:/data"
    depends_on:
      - resolver
    dns:
      - 172.69.69.254

  postgres:
    image: postgres:11
    container_name: "mailu-db"
    restart: always
    environment:
      - PUID=100
      - PGID=1000
      - TZ=Europe/Rome
      # Postgres
      - POSTGRES_USER=mailu
      - POSTGRES_PASSWORD=xyz
      - POSTGRES_DB=mailu
    networks:
      - mailu
    volumes:
      - "mailu-postgres:/var/lib/postgresql/data"
    depends_on:
      - resolver
    dns:
      - 172.69.69.254

  pgadmin:
    container_name: "mailu-pgadmin"
    image: dpage/pgadmin4
    restart: always
    environment:
      PGADMIN_DEFAULT_EMAIL: admin@admin.com
      PGADMIN_DEFAULT_PASSWORD: password
    networks:
      - mailu
    ports:
      - "5050:80"

  # Core services
  front:
    container_name: "mailu-front"
    # image: "mailu/nginx:1.9"
    image: "hitech95/mailu-nginx:1.9-feat-haproxy"
    restart: always
    environment:
      - PUID=100
      - PGID=1000
      - TZ=Europe/Rome
      # NGINX
      - HOSTNAMES=mail.mydomain.com
      - KUBERNETES_INGRESS=true
      - REAL_IP_HEADER=X-Forwarded-For
      - REAL_IP_FROM=172.77.77.254
      - EMAIL_PROXYED=true
      - PROXY_PROTOCOL=true
      - WEBROOT_REDIRECT=none
      - WEB_ADMIN=/admin
      - ADMIN=true
      - WEBMAIL=none
      - WEBDAV=none
      - MESSAGE_SIZE_LIMIT=500000000
      # START TLS
      - "TLS_FLAVOR=mail"
      - "TLS_CERT_FILENAME=mail.mydomain.com/cert.pem"
      - "TLS_KEYPAIR_FILENAME=mail.mydomain.com/key.pem"
    labels:
      # WEB UI
      - "traefik.enable=true"
      - "traefik.http.services.mailu-web.loadbalancer.server.port=80"
      - "traefik.http.services.mailu-web.loadbalancer.server.scheme=http"
      - "traefik.http.routers.mailu-web.rule=Host(`mail.mydomain.com`)"
      - "traefik.http.routers.mailu-web.entrypoints=web,websecure"
      - "traefik.http.routers.mailu-web.tls=true"
      - "traefik.http.routers.mailu-web.tls.certresolver=letsencrypt"
      - "traefik.http.routers.mailu-web.tls.domains[0].main=mail.mydomain.com"
      - "traefik.http.routers.mailu-web.tls.domains[0].sans=imaps.mydomain.com,imap.mydomain.com,pop3s.mydomain.com,pop3.mydomain.com,smtps.gremydomain.com,smtp.mydomain.com"
      - "traefik.http.routers.mailu-web.service=mailu-web"
      # EMAIL
      # SMTP Relay (START TLS)
      - "traefik.tcp.services.mailu-smtp.loadbalancer.server.port=25"
      - "traefik.tcp.services.mailu-smtp.loadbalancer.proxyProtocol=true"
      - "traefik.tcp.routers.mailu-smtp.rule=HostSNI(`mail.mydomain.com`, `smtp.mydomain.com`, `*`)"
      - "traefik.tcp.routers.mailu-smtp.entrypoints=smtp"
      - "traefik.tcp.routers.mailu-smtp.tls=false"
      - "traefik.tcp.routers.mailu-smtp.service=mailu-smtp"
      # SMTP Submission (START TLS)
      # - "traefik.tcp.services.mailu-submission.loadbalancer.server.port=587"
      # - "traefik.tcp.services.mailu-submission.loadbalancer.proxyProtocol=true"
      # - "traefik.tcp.routers.mailu-submission.rule=HostSNI(`mail.mydomain.com`, `smtp.mydomain.com`, `*`)"
      # - "traefik.tcp.routers.mailu-submission.entrypoints=smtp-auth"
      # - "traefik.tcp.routers.mailu-submission.tls=false"
      # - "traefik.tcp.routers.mailu-submission.service=mailu-submission"
      # SMTP Submission (SSL)
      - "traefik.tcp.services.mailu-smtps.loadbalancer.server.port=465"
      - "traefik.tcp.services.mailu-smtps.loadbalancer.proxyProtocol=true"
      - "traefik.tcp.routers.mailu-smtps.rule=HostSNI(`mail.mydomain.com`, `smtps.mydomain.com`)"
      - "traefik.tcp.routers.mailu-smtps.entrypoints=smtps"
      - "traefik.tcp.routers.mailu-smtps.tls.passthrough=true"
      - "traefik.tcp.routers.mailu-smtps.service=mailu-smtps"
      # IMAP (START TLS)
      # - "traefik.tcp.services.mailu-imap.loadbalancer.server.port=143"
      # - "traefik.tcp.services.mailu-imap.loadbalancer.proxyProtocol=true"
      # - "traefik.tcp.routers.mailu-imap.rule=HostSNI(`mail.mydomain.com`, `imap.mydomain.com`, `*`)"
      # - "traefik.tcp.routers.mailu-imap.entrypoints=imap"
      # - "traefik.tcp.routers.mailu-imap.passthrough=true"
      # - "traefik.tcp.routers.mailu-imap.service=mailu-imap"
      # IMAP (SSL)
      - "traefik.tcp.services.mailu-imaps.loadbalancer.server.port=993"
      - "traefik.tcp.services.mailu-imaps.loadbalancer.proxyProtocol=true"
      - "traefik.tcp.routers.mailu-imaps.rule=HostSNI(`mail.mydomain.com`, `imaps.mydomain.com`)"
      - "traefik.tcp.routers.mailu-imaps.entrypoints=imaps"
      - "traefik.tcp.routers.mailu-imaps.tls.passthrough=true"
      - "traefik.tcp.routers.mailu-imaps.service=mailu-imaps"
      # POP3 (START TLS)
      # - "traefik.tcp.services.mailu-pop3.loadbalancer.server.port=110"
      # - "traefik.tcp.services.mailu-pop3.loadbalancer.proxyProtocol=true"
      # - "traefik.tcp.routers.mailu-pop3.rule=HostSNI(`mail.mydomain.com`, `pop3.mydomain.com`, `*`)"
      # - "traefik.tcp.routers.mailu-pop3.entrypoints=pop"
      # - "traefik.tcp.routers.mailu-pop3.tls=false"
      # - "traefik.tcp.routers.mailu-pop3.service=mailu-pop3"
      # POP3 (SSL)
      # - "traefik.tcp.services.mailu-pop3s.loadbalancer.server.port=11110"
      # - "traefik.tcp.services.mailu-pop3s.loadbalancer.proxyProtocol=true"
      # - "traefik.tcp.routers.mailu-pop3s.rule=HostSNI(`mail.mydomain.com`, `pop3s.mydomain.com`)"
      # - "traefik.tcp.routers.mailu-pop3s.entrypoints=pops"
      # - "traefik.tcp.routers.mailu-pop3s.passthrough=true"
      # - "traefik.tcp.routers.mailu-pop3s.service=mailu-pop3s"
    networks:
      - mailu
      - proxy
    volumes:
      - "mailu-front:/overrides:ro"
      - "mailu-certs:/certs:ro"
    depends_on:
      - resolver
      - admin
    dns:
      - 172.69.69.254

  admin:
    container_name: "mailu-admin"
    image: mailu/admin:1.9
    restart: always
    environment:
      - PUID=100
      - PGID=1000
      - TZ=Europe/Rome
      # App
      - DISABLE_STATISTICS=true
      - SECRET_KEY=xyz
      - RECIPIENT_DELIMITER=+
      # Network
      - SUBNET=172.69.69.0/24
      - DOMAIN=mydomain.com
      - HOSTNAMES=mail.mydomain.com,servername.mydomain.com
      - POSTMASTER=admin
      # Auth
      - AUTH_RATELIMIT_IP=60/hour
      - AUTH_RATELIMIT_USER=100/day
      # DB Connection
      - DB_FLAVOR=postgresql
      - DB_HOST=postgres
      - DB_USER=mailu
      - DB_PW=mypassword
      - DB_NAME=mailu
    networks:
      - mailu
    volumes:
      - "mailu-admin:/data"
      - "mailu-dkim:/dkim"
    depends_on:
      - redis
      - postgres
    dns:
      - 172.69.69.254

  imap:
    container_name: "mailu-imap"
    image: mailu/dovecot:1.9
    restart: always
    environment:
      - PUID=100
      - PGID=1000
      - TZ=Europe/Rome
      # DOVECOT
      - DOMAIN=mydomain.com
      - HOSTNAMES=mail.mydomain,servername.mydomain.com
      - SUBNET=172.69.69.0/24
      - POSTMASTER=admin
      - RECIPIENT_DELIMITER=+
    networks:
      - mailu
    volumes:
      - "mailu-imap:/mail"
    depends_on:
      - resolver
      - front
    dns:
      - 172.69.69.254

  smtp:
    container_name: "mailu-smtp"
    image: mailu/postfix:1.9
    restart: always
    environment:
      - PUID=100
      - PGID=1000
      - TZ=Europe/Rome
      # POSTFIX
      - DOMAIN=mydomain.com
      - HOSTNAMES=mail.mydomain.com,servername.mydomain.com
      - SUBNET=172.69.69.0/24
      - RELAYNETS=172.70.70.0/24
      - RECIPIENT_DELIMITER=+
      - MESSAGE_SIZE_LIMIT=500000000
      - MESSAGE_RATELIMIT=200/day
    networks:
      - mailu
    volumes:
      - "mailu-smtp:/queue"
    depends_on:
      - resolver
      - front
    dns:
      - 172.69.69.254

  antispam:
    container_name: "mailu-antispam"
    image: mailu/rspamd:1.9
    restart: always
    environment:
      - PUID=100
      - PGID=1000
      - TZ=Europe/Rome
      # RSPAMD
      - SUBNET=172.69.69.0/24
      - RELAYNETS=172.70.70.0/24
      - ANTIVIRUS=clamav
    networks:
      - mailu
    volumes:
      - "mailu-rspamd:/var/lib/rspamd"
      - "mailu-dkim:/dkim"
    depends_on:
      - resolver
      - front
    dns:
      - 172.69.69.254

An example of my stack that use local relay:

version: '3'
networks:
  web:
    external: true
    name: proxy
  relay:
    external: true
    name: relay
  vaultwarden:
    external: false
    name: vaultwarden
    
volumes:
  bitwarden:
    external: true
    name: bitwarden
  postgres:
    external: true
    name: bitwarden_db
    
services:
  postgres:
    image: postgres:11
    container_name: vaultwarden_pgsql
    restart: always
    environment:
      - PUID=100
      - PGID=1000
      - TZ=Europe/Rome
      - POSTGRES_USER=vaultwarden
      - POSTGRES_PASSWORD=xyz
      - POSTGRES_DB=bitwarden
    networks:
      - vaultwarden
    volumes:
      - postgres:/var/lib/postgresql/data
      
  vaultwarden:
    image: vaultwarden/server:latest
    container_name: vaultwarden
    restart: always
    environment:
      - PUID=100
      - PGID=1000
      - TZ=Europe/Rome
      # Vaultwarden
      - WEBSOCKET_ENABLED=true  # Enable WebSocket notifications.
      - DATABASE_URL=postgresql://vaultwarden:xyz@postgres/bitwarden
      - ADMIN_TOKEN=xyz
    networks:
      - web
      - relay
      - vaultwarden
    labels:
      - "traefik.enable=true"
      # Two services are running
      # See: https://github.com/SergioBenitez/Rocket/issues/90
      # And: https://github.com/dani-garcia/vaultwarden/wiki/Enabling-WebSocket-notifications
      - "traefik.http.services.vaultwardenUI.loadbalancer.server.port=80"
      - "traefik.http.services.vaultwardenSocket.loadbalancer.server.port=3012"      
      # Route for UI
      - "traefik.http.routers.vaultwardenUI.rule=Host(`bitwarden.mydomain.com`)"
      - "traefik.http.routers.vaultwardenUI.entrypoints=websecure"
      - "traefik.http.routers.vaultwardenUI.tls.certresolver=letsencrypt"
      - "traefik.http.routers.vaultwardenUI.service=vaultwardenUI"
      # Route for Websockets
      - "traefik.http.routers.vaultwardenSocket.rule=Host(`bitwarden.mydomain.com`) && Path(`/notifications/hub`)"
      - "traefik.http.routers.vaultwardenSocket.entrypoints=websecure"
      - "traefik.http.routers.vaultwardenSocket.tls.certresolver=letsencrypt"      
      - "traefik.http.routers.vaultwardenSocket.service=vaultwardenSocket"
    volumes:
      - bitwarden:/data
    depends_on:
      - postgres

Local relay settings are: host: 172.70.70.254 port 25 no auth

He's not needed to map mail ports to traefik ? Like 25 587 etc...

[t0mtaylor]

any update on this? would be good to have resolved and merged in 👍

[ghostwheel42]

any update on this? would be good to have resolved and merged in 👍

You'll have to wait for the next release or start using master (our development branch) to use it.
When you switch to master you can also just use this PRs images to test it and give feedback.

[hitech95]

Original patch author here.
I'm not sure how you guys want to handle this.

I didnt had any new feedback.
So I'm dropping an example to discuss, but before that I want to focus on this caviat:
As stated before not all proxies can do STARTLS

Example network is composed as:

Internet < --- > Proxy (traefik) < -- HAPROXY -- > Front
                                   email_subnet  <-^
Internal Service 1            < ---^
Internal Service 2            < ---^
Internal Service 3            < ---^
Bitwarden                     < ---^

How would you handle this case with the proxy protocol enabled?

• How would you handle the STARTLS? cosnsidering that:
  • Most proxies can do TLS termination or TLS pass through.
  • AFIK only nginx can do STARTLS, so front MUST handle STARTLS.
    My solution was to use TLS pass through and front manage TLS/STARTLS termination.

This expose another issue/problem:

• Should you route the internal traffic via the proxy to encapsulate is in the HA_PROXY?
  • If NO you need an additional port set for local trusted connections probably bypassing HA Proxy and SSL.
  • If YES then you need to also have your border proxy to listen to internal networks.

While developing the patch I've also discovered the following user case that I tried to mitigate using EMAIL_PROXYED:

• In the internal email_subnet is it really necessary to have SSL/TLS?

  • If NOT then you should pass via your border proxy if you have HA Proxy enabled on a port set without SSL
    but this is not possible, due to SSL termination handled by front. if no HA Proxy it is the same as above.
    You might need an additional portset in the front container.

  • If YES then you need a way to directly connect to a portset that does not need HA Proxy
    and your services must be connected to internet to validate if the certs are valid. Or you go throu the border proxy.

[pr0ton11]

I also have a use case that requires this if I want to run it in k8s:

HAProxy (terminates TLS, but also IPv4) -> Nginx Ingress Controller (IPv6 only) -> Mailu (IPv6 only).

At the moment I have to use a dedicated VM running a docker setup. If this gets merged I can migrate it to K8s and use HA, Distributed Storage etc.
It doesn't matter for me if I need to provide the certs for mail separately (I can do that in a k8s secret)

[t0mtaylor]

I'm running with a DO LB (in proxy protocol mode), to Traefik, then to Mailu via Docker Swarm.

Traffic in -> DO LB -> [Server] (Docker Swarm) Global -> Traefik Router-> Mailu Frontend Nginx -> Mailu Services

Having run my setup for over a year, this is the final hurdle, and here is some research I have conducted.

Now its a longshot, but it appears theres an issue with the Docker overlay network (when you create a swarm), which it seems does not allow for ip forwarding - https://community.traefik.io/t/whitelist-swarm-cant-get-real-source-ip/3897/2

I've listed a couple of options here to try, which may help with the Proxy Protocol support - i'm not sure if PP may need to be turned off in these test cases below but its worth trying!

There is also an indepth post about [Unable to retrieve user's IP address in docker swarm mode](https://github.com/moby/moby/issues/25526#top) - moby/moby#25526
One way around this is to not use overlay and use the host directive and ports direct:

https://stackoverflow.com/a/44648488 - details that you would run Traefik on each server in your swarm as global deployment to map to the ports on each server
moby/moby#25526 (comment)

 ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host

There is also the docker-ingress-routing-daemon - https://github.com/newsnowlabs/docker-ingress-routing-daemon which looks promising and may be worth a try, which reads:

Docker Swarm's out-of-the-box ingress mesh routing logic uses IPVS and SNAT to route incoming traffic to service containers. By using SNAT to masquerade the source IP of each incoming connection to be the ingress network IP of the load balancer node, service containers receiving traffic from multiple load balancer nodes are able to route the reverse path traffic back to the correct node (which is necessary for the SNAT to be reversed and the reverse path traffic returned to the correct client IP).

An unfortunate side-effect of this approach is that to service containers, all incoming traffic appears to arrive from the same set of private network ingress network node IPs, meaning service containers cannot distinguish individual clients by IP, or geolocate clients.

This has been documented in moby/moby issue moby/moby#25526 (as well as moby/moby#15086, moby/moby#26625 and moby/moby#27143).

Typical existing workarounds require running an independent reverse-proxy load-balancer, like nginx or traefik, in front of your docker services, and modifying your applications to examine the X-Forwarded-For header. Compared to docker's own load balancer, which uses the kernel's IPVS, this is likely to be less efficient.

This may be worth trying as it seems to tackle the common issue we are having

I've also seen some posts around enabling IPV6 for Docker Swarm due to overlay issues, which means you need to recreate this network bridge for the swarm - https://github.com/robbertkl/docker-ipv6nat - although with latest version of docker this container may no longer be required - robbertkl/docker-ipv6nat#65 - but the recreation to allow for ip forwarding whilst keeping the overlay network is interesting!

With this in mind, I've modified this gist with the ipv6 swarm mode config - with the additional ip forwarding - i've not tested this yet so please text on a non-production environment if you wish to try

https://gist.github.com/hgross/26042d052f58feeb6d1b329e8dd2dfcc (this was to change subnet address)
https://github.com/robbertkl/docker-ipv6nat (see Swarm mode support)

You can check existing network setup here on your server node:

docker network ls --quiet | xargs docker network inspect --format '{{ .Name }}: {{ .Options }}' 

Heres the bash steps, which will vary if your running docker as a service for your swarm, etc.

## Do this on each swarm-node
# store containers attached to the bridge, useful for debugging
$ gwbridge_users=$(docker network inspect --format '{{range $key, $val := .Containers}} {{$key}}{{end}}' docker_gwbridge | \
$ xargs -d' ' -I {} -n1 docker ps --format {{.Names}} -f id={})


# if you have stacks that restart automatically, remove them via docker stack <stackName> rm

# if you run via a service, drain the node first then stop the docker service
docker swarm leave –-force

# Remove node from the docker swarm once drained before you stop the service
echo "$gwbridge_users" | xargs docker stop

docker network rm docker_gwbridge
docker network disconnect -f docker_gwbridge gateway_ingress-sbox
docker network rm docker_gwbridge

SUBNET=172.20.0.0/20
GATEWAY=172.20.0.1

docker network create  \
--ipv6 \ #if you want it
--subnet=${SUBNET}     \
--gateway ${GATEWAY}   \
 --gateway fd00:3984:3989::1 \ #ipv6
 --subnet fd00:3984:3989::/64 \ #ipv6
 --opt com.docker.network.bridge.name=docker_gwbridge \
 --opt com.docker.network.bridge.enable_icc=true \
 --opt com.docker.network.bridge.enable_ip_forwarding=true \
 --opt com.docker.network.bridge.enable_ip_masquerade=true \
docker_gwbridge

#if you running docker via a service, you may not need to do this and just restart the service
echo "$gwbridge_users" | xargs docker start

#Rejoin your docker swarm and manager or worker, and test!

Hope this helps so we can finally bring the client ips into our containers!

[OdyX]

This is clearly introducing a bug; the trailing semicolon is mandatory.