Trusted Proxy Network: when set, UI doesn't load

[jayknyn]

Mauro, thanks for your prompt replies! 🍻

I've set up Stalwart-mail v0.7.0 in Docker with Postgres as the default store, proxied behind Traefik with Proxy Protocol 2. TLS certs have been extracted from Traefik's ACME store and placed in the Stalwart container.

When I set the trusted proxy network to the IP of my Traefik container, I get a 502 Bad Gateway from the UI with the logs showing invalid proxy header. However, with this trusted proxy network set, Implicit TLS for port 465 and 993 works as expected from Roundcube and OpenSSL.

When I remove the trusted proxy network, the UI loads but the TCP proxy breaks for all other listeners. Do you have any suggestions on how to resolve this?

Relevant Stalwart logs (172.18.0.10 = IP of Traefik):

With trusted proxy network not set (UI loads, other listeners fail):

stalwart-mail  | 2024-04-13T00:51:07.316695Z DEBUG session{instance="http" protocol=Http remote.ip="172.18.0.10" remote.port=41120}: jmap::api::http: event="request" uri="/favicon-32x32-33abe4bf0e53934c.png"
stalwart-mail  | 2024-04-13T00:51:20.534971Z DEBUG common::listener: Failed to accept TLS connection: received corrupt message of type InvalidContentType context="tls" event="error" instance="submissions" protocol=Smtp remote.ip="172.18.0.10"

With trusted proxy network set (UI doesn't load, other listeners succeed):

stalwart-mail  | 2024-04-13T00:53:27.154779Z TRACE common::listener::listen: Failed to accept proxied TCP connection context="io" event="error" instance="http" protocol=Http reason=invalid proxy header

stalwart-mail  | 2024-04-13T00:53:42.401704Z TRACE common::expr::eval: context="eval_if" property="session.connect.greeting" result=String("Stalwart ESMTP at your service")
stalwart-mail  | 2024-04-13T00:53:42.402402Z TRACE session{instance="submissions" protocol=Smtp remote.ip="x.x.x.x" remote.port=35780}: smtp::inbound::session: event="write" data="220 Stalwart ESMTP at your service\r\n" size=36

Relevant parts of config.toml:

certificate.traefik.cert = "%{file:/opt/stalwart-mail/certs/mail.domain.tld.crt}%"
certificate.traefik.private-key = "%{file:/opt/stalwart-mail/certs/mail.domain.tld.key}%"
certificate.traefik.default = true
lookup.default.hostname = "mail.domain.tld"
server.http.permissive-cors = false
server.http.url = "protocol + '://' + key_get('default', 'hostname') + ':' + local_port"
server.http.use-x-forwarded = false
server.listener.http.bind = "[::]:8080"
server.listener.http.protocol = "http"
server.listener.https.bind = "[::]:443"
server.listener.https.protocol = "http"
server.listener.https.tls.implicit = true
server.listener.imaptls.bind = "[::]:993"
server.listener.imaptls.protocol = "imap"
server.listener.imaptls.tls.implicit = true
server.listener.sieve.bind = "[::]:4190"
server.listener.sieve.protocol = "managesieve"
server.listener.sieve.tls.implicit = true
server.listener.smtp.bind = "[::]:25"
server.listener.smtp.protocol = "smtp"
server.listener.submission.bind = "[::]:587"
server.listener.submission.protocol = "smtp"
server.listener.submissions.bind = "[::]:465"
server.listener.submissions.protocol = "smtp"
server.listener.submissions.tls.implicit = true
server.max-connections = 8192
# server.proxy.trusted-networks = "172.18.0.0/16"
server.socket.backlog = 1024
server.socket.nodelay = true
server.socket.reuse-addr = true
server.socket.reuse-port = true
server.tls.certificate = "traefik"
server.tls.enable = true

Relevant config:

Stalwart-mail docker-compose.yml

version: '3.7'

networks:
ingress:
  external: true

default:
  internal: true
  name: stalwart

services:
stalwart:
  container_name: ${SERVICE_NAME}-mail
  image: stalwartlabs/mail-server:${SERVICE_VERSION} # arm and x86
  depends_on:
    db:
      condition: service_healthy
  env_file: .env
  networks:
    ingress:
      ipv4_address: '172.18.0.20'
    default:
  security_opt: [no-new-privileges:true]
  restart: unless-stopped
  # healthcheck:
  labels:
    - traefik.enable=true
    # admin ui
    - traefik.http.routers.stalwart.rule=Host(`${DOMAIN}`)
    - traefik.http.routers.stalwart.entrypoints=websecure
    - traefik.http.routers.stalwart.tls.certresolver=le
    - traefik.http.routers.stalwart.service=stalwart
    - traefik.http.services.stalwart.loadbalancer.server.port=8080
    # jmap
    - traefik.tcp.routers.jmap.rule=HostSNI(`*`)
    - traefik.tcp.routers.jmap.entrypoints=websecure
    - traefik.tcp.routers.jmap.tls.passthrough=true
    - traefik.tcp.routers.jmap.service=jmap
    - traefik.tcp.services.jmap.loadbalancer.server.port=443
    - traefik.tcp.services.jmap.loadbalancer.proxyProtocol.version=2
    # smtp
    - traefik.tcp.routers.smtp.rule=HostSNI(`*`)
    - traefik.tcp.routers.smtp.entrypoints=smtp
    - traefik.tcp.routers.smtp.service=smtp
    - traefik.tcp.services.smtp.loadbalancer.server.port=25
    - traefik.tcp.services.smtp.loadbalancer.proxyProtocol.version=2
    # esmtp
    - traefik.tcp.routers.esmtp.rule=HostSNI(`*`)
    - traefik.tcp.routers.esmtp.entrypoints=esmtp
    - traefik.tcp.routers.esmtp.tls.passthrough=true
    - traefik.tcp.routers.esmtp.service=esmtp
    - traefik.tcp.services.esmtp.loadbalancer.server.port=465
    - traefik.tcp.services.esmtp.loadbalancer.proxyProtocol.version=2
    # imap
    - traefik.tcp.routers.imap.rule=HostSNI(`*`)
    - traefik.tcp.routers.imap.entrypoints=imap
    - traefik.tcp.routers.imap.tls.passthrough=true
    - traefik.tcp.routers.imap.service=imap
    - traefik.tcp.services.imap.loadbalancer.server.port=993
    - traefik.tcp.services.imap.loadbalancer.proxyProtocol.version=2
    # sieve
    - traefik.tcp.routers.sieve.rule=HostSNI(`*`)
    - traefik.tcp.routers.sieve.entrypoints=sieve
    - traefik.tcp.routers.sieve.tls.passthrough=true
    - traefik.tcp.routers.sieve.service=sieve
    - traefik.tcp.services.sieve.loadbalancer.server.port=4190
    - traefik.tcp.services.sieve.loadbalancer.proxyProtocol.version=2
  volumes:
    - ./data:/opt/stalwart-mail
  logging:
    driver: 'json-file'
    options:
      max-size: '200k'
      max-file: '10'
  # ports:
  #   - '25:25'
  #   - '443:443'
  #   - '465:465'
  #   # - "587:587" # not using
  #   - '993:993'
  #   - '4190:4190'
  #   - '8080:8080' # admin ui

db:
  container_name: ${SERVICE_NAME}-db
  image: postgres:${DB_VERSION}
  environment:
    - POSTGRES_DB=${DB_NAME}
    - POSTGRES_USER=${DB_USER}
    - POSTGRES_PASSWORD=${DB_PASSWORD}
    - TZ
  networks: [default]
  restart: unless-stopped
  healthcheck:
    test: ['CMD', 'pg_isready', '-q', '-d', '${DB_NAME}', '-U', '${DB_USER}']
    timeout: 45s
    interval: 10s
    retries: 10
  volumes: [./db:/var/lib/postgresql/data]

Stalwart-mail full etc/config.toml

authentication.fallback-admin.secret = "SECRET"
authentication.fallback-admin.user = "admin"
certificate.traefik.cert = "%{file:/opt/stalwart-mail/certs/mail.domain.tld.crt}%"
certificate.traefik.private-key = "%{file:/opt/stalwart-mail/certs/mail.domain.tld.key}%"
certificate.traefik.default = true
cluster.node-id = 1
directory.postgres.cache.entries = 500
directory.postgres.cache.ttl.negative = "10m"
directory.postgres.cache.ttl.positive = "1h"
directory.postgres.columns.class = "type"
directory.postgres.columns.email = "address"
directory.postgres.columns.description = "description"
directory.postgres.columns.name = "name"
directory.postgres.columns.quota = "quota"
directory.postgres.columns.secret = "secret"
directory.postgres.store = "postgres"
directory.postgres.type = "sql"
lookup.default.hostname = "mail.domain.tld"
server.http.permissive-cors = false
server.http.url = "protocol + '://' + key_get('default', 'hostname') + ':' + local_port"
server.http.use-x-forwarded = false
server.listener.http.bind = "[::]:8080"
server.listener.http.protocol = "http"
server.listener.https.bind = "[::]:443"
server.listener.https.protocol = "http"
server.listener.https.tls.implicit = true
server.listener.imaptls.bind = "[::]:993"
server.listener.imaptls.protocol = "imap"
server.listener.imaptls.tls.implicit = true
server.listener.sieve.bind = "[::]:4190"
server.listener.sieve.protocol = "managesieve"
server.listener.sieve.tls.implicit = true
server.listener.smtp.bind = "[::]:25"
server.listener.smtp.protocol = "smtp"
server.listener.submission.bind = "[::]:587"
server.listener.submission.protocol = "smtp"
server.listener.submissions.bind = "[::]:465"
server.listener.submissions.protocol = "smtp"
server.listener.submissions.tls.implicit = true
server.max-connections = 8192
server.proxy.trusted-networks = "172.18.0.0/16"
server.socket.backlog = 1024
server.socket.nodelay = true
server.socket.reuse-addr = true
server.socket.reuse-port = true
server.tls.certificate = "traefik"
server.tls.enable = true
storage.blob = "r2"
storage.data = "postgres"
storage.directory = "postgres"
storage.encryption.enable = true
storage.encryption.append = false
storage.fts = "postgres"
storage.lookup = "postgres"
store.postgres.compression = "lz4"
store.postgres.database = "stalwart"
store.postgres.host = "db"
store.postgres.password = "SECRET"
store.postgres.pool.max-connections = 10
store.postgres.port = 5432
store.postgres.purge.frequency = "0 3 *"
store.postgres.query.domains = "SELECT 1 FROM emails WHERE address LIKE '%@' || $1 LIMIT 1"
store.postgres.query.emails = "SELECT address FROM emails WHERE name = $1 AND type != 'list' ORDER BY type DESC, address ASC"
store.postgres.query.expand = "SELECT p.address FROM emails AS p JOIN emails AS l ON p.name = l.name WHERE p.type = 'primary' AND l.address = $1 AND l.type = 'list' ORDER BY p.address LIMIT 50"
store.postgres.query.members = "SELECT member_of FROM group_members WHERE name = $1"
store.postgres.query.name = "SELECT name, type, secret, description, quota FROM accounts WHERE name = $1 AND active = true"
store.postgres.query.recipients = "SELECT name FROM emails WHERE address = $1 ORDER BY name ASC"
store.postgres.query.verify = "SELECT address FROM emails WHERE address LIKE '%' || $1 || '%' AND type = 'primary' ORDER BY address LIMIT 5"
store.postgres.timeout = "15s"
store.postgres.tls.allow-invalid-certs = false
store.postgres.tls.enable = false
store.postgres.type = "postgresql"
store.postgres.user = "stalwart"
store.r2.access-key = "SECRET"
store.r2.bucket = "stalwart"
store.r2.endpoint = "https://SECRET.r2.cloudflarestorage.com"
store.r2.purge.frequency = "0 3 *"
store.r2.region = "auto"
store.r2.secret-key = "SECRET"
store.r2.timeout = "15s"
store.r2.type = "s3"
tracer.stdout.ansi = true
tracer.stdout.enable = true
tracer.stdout.level = "debug"
tracer.stdout.type = "stdout"

Traefik docker-compose.yml

version: '3.7'

networks:
# docker network create ingress --subnet "172.18.0.0/24" --gateway "172.18.0.1"
ingress:
  external: true
default:
  internal: true
  name: traefik

services:
traefik:
  container_name: traefik
  image: traefik:${TRAEFIK_VERSION}
  depends_on: [dockerproxy, watchtower]
  networks:
    ingress:
      ipv4_address: '172.18.0.10'
    default:
  security_opt: [no-new-privileges:true]
  ports:
    - '80:80'
    - '443:443'
    # Ports for MailServer
    - '25:25' # SMTP receiving, explicit TLS
    - '465:465' # ESMTP submission, implicit TLS
    - '993:993' # IMAP4 secure, implicit TLS
    - '4190:4190' # sieve
  restart: always
  command:
    - --log.level=DEBUG # DEBUG, PANIC, FATAL, ERROR, WARN, INFO
    - --global.checknewversion=false
    - --global.sendAnonymousUsage=false
    # Activate dashboard
    - --api=true
    # Activate /ping healthcheck endpoint
    - --ping=true
    - --ping.manualRouting=true
    - --ping.entrypoint=websecure
    # Activate docker provider, but do not expose every container to traefik
    - --providers.docker=true
    - --providers.docker.exposedbydefault=false
    - --providers.docker.endpoint=tcp://dockerproxy:2375
    - --providers.docker.network=ingress
    # HTTPS entrypoints and redirect
    - --entrypoints.webinsecure.address=:80
    - --entrypoints.webinsecure.http.redirections.entrypoint.to=websecure
    - --entrypoints.webinsecure.http.redirections.entrypoint.scheme=https
    - --entrypoints.websecure.address=:443
    # TCP entrypoints for MailServer
    - --entrypoints.smtp.address=:25
    - --entrypoints.esmtp.address=:465
    - --entrypoints.imap.address=:993
    - --entrypoints.sieve.address=:4190
    # DNS Challenge with Cloudflare 
    - --certificatesresolvers.le.acme.dnschallenge.provider=cloudflare
    - --certificatesresolvers.le.acme.dnschallenge.resolvers=1.1.1.1:53,9.9.9.9:53
    - --certificatesresolvers.le.acme.email=${LETS_ENCRYPT_EMAIL}
    - --certificatesresolvers.le.acme.storage=/certs/acme.json
    # Let's Encrypt Staging Server, start with staging then switch to prod server
    #- --certificatesResolvers.le.acme.caServer=https://acme-staging-v02.api.letsencrypt.org/directory
    # Let's Encrypt Production Server
    - --certificatesResolvers.le.acme.caServer=https://acme-v02.api.letsencrypt.org/directory
  environment:
    - CLOUDFLARE_DNS_API_TOKEN
  labels:
    - com.centurylinklabs.watchtower.enable=true
    - traefik.enable=true
    # Traefik Dashboard available at $DASHBOARD_DOMAIN/dashboard/ (trailing slash required)
    - traefik.http.routers.dashboard.rule=Host(`${DASHBOARD_DOMAIN}`)
    - traefik.http.routers.dashboard.entrypoints=websecure
    - traefik.http.routers.dashboard.tls.certresolver=le
    - traefik.http.routers.dashboard.service=api@internal
    # - traefik.http.routers.dashboard.middlewares=secure-headers@file
    - traefik.http.routers.dashboard.middlewares=dashboard-auth
    - traefik.http.middlewares.dashboard-auth.basicauth.usersfile=/.env.users
    # Routers for /ping healthcheck endpoint
    - traefik.http.routers.ping.rule=Host(`${DASHBOARD_DOMAIN}`) && Path(`/ping`)
    - traefik.http.routers.ping.entrypoints=websecure
    - traefik.http.routers.ping.tls.certresolver=le
    - traefik.http.routers.ping.service=ping@internal
  volumes:
    - ./certs:/certs
    - ./.env.users:/.env.users:ro # add user:hashed-password to .env.users

dockerproxy:
  container_name: traefik-dockerproxy
  image: ghcr.io/tecnativa/docker-socket-proxy:${DOCKER_SOCKET_PROXY_VERSION}
  depends_on: [watchtower]
  environment:
    CONTAINERS: 1
  networks: [default]
  ports: [2375]
  restart: always
  labels:
    - com.centurylinklabs.watchtower.enable=true
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock:ro

watchtower:
  container_name: traefik-watchtower
  image: containrrr/watchtower:${WATCHTOWER_VERSION}
  command: --label-enable --cleanup --interval 86400
  network_mode: none
  restart: always
  labels:
    - com.centurylinklabs.watchtower.enable=true
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock

[mdecimus]

Try accessing the UI through your proxy. You could be accessing Stalwart using an IP address that is in the same proxy-networks IP range and Stalwart is expecting to receive the proxy protocol headers which your browser won't provide.
If that is not the case (i.e. your request is not coming from one of the IPs defined in proxy-networks), then check the Stalwart logs to see what went wrong.

[jayknyn]

Thanks for your response, Mauro. So, I am accessing the UI through my proxy (Traefik).

https://mail.domain.tld -> Traefik Proxy (172.18.0.10) -> Stalwart container (172.18.0.20)

Traefik logs:

level=debug msg="'502 Bad Gateway' caused by: read tcp 172.18.0.10:36026->172.18.0.20:8080: read: connection reset by peer"

And Stalwart logs on Trace show:

TRACE common::listener::listen: Failed to accept proxied TCP connection context="io" event="error" instance="http" protocol=Http reason=invalid proxy header

I looked in the code and this error message is coming from ln 145 in src/listener/listen.rs

I'm not at all proficient in Rust, but from this block in src/config/server/listerner.rs, is it the case that if server.proxy.trusted-networks has a value, then all the listeners expect their traffic to come from this trusted proxy network? If that's the case, then I think the webadmin won't work as is with proxies like Traefik. My understanding is that Traefik automatically relays the incoming traffic to the mapped router.

Do you think it's possible for the webadmin (8080) not to be listening for its HTTP traffic behind the trusted proxy while the other TCP listeners do if the trusted proxy is defined in the config?

For the other TCP listeners (smtp, imap, sieve), I'm using proxy protocol and that's working as expected, the headers are arriving as Stalwart expects. And when I remove the trusted proxy from the config.toml, the webadmin is correctly served up by Traefik.

Stalwart logs with trusted proxy disabled:

INFO common::listener::listen: Starting listener id="http" protocol=Http bind.ip="::" bind.port=8080 tls=false
DEBUG session{instance="http" protocol=Http remote.ip="172.18.0.10" remote.port=41246}: jmap::api::http: event="request" uri="/"
DEBUG session{instance="http" protocol=Http remote.ip="172.18.0.10" remote.port=41246}: jmap::api::http: event="request" uri="/output-86e4f49e40c73dc8.css"

[mdecimus]

This error message you provided indicates that Traefik is not sending the proxy headers to Stalwart:

TRACE common::listener::listen: Failed to accept proxied TCP connection context="io" event="error" instance="http" protocol=Http reason=invalid proxy header

Here is a sample Traefik configuration file that another user contributed:

networks:
###
  stalwart:
    driver: bridge
    attachable: true
    internal: true
    driver_opts:
      com.docker.network.driver.mtu: 9000
###
services:
###
  traefik:
###
    container_name: traefik
    read_only: true
    tmpfs:
      - /run
      - /var/run
    command:
###
      - "--entrypoints.smtpsecure.address=:465"
      - "--entrypoints.imapsecure.address=:993"
###
      - "--providers.docker"
###
    ports:
      - "465:465/tcp"
      - "993:993/tcp"
    networks:
      - internet
      - stalwart
###
  stalwart:
###
    container_name: stalwart
    read_only: true
    tmpfs:
      - /run
      - /var/run
###
    ports:
      - "25:25/tcp"
      - "587:587/tcp"
###
    labels:
###
      - "traefik.enable=true"
      - "traefik.tcp.routers.smtpsecure.tls=true"
      - "traefik.tcp.routers.smtpsecure.tls.certresolver=letsencrypt"
      - "traefik.tcp.routers.smtpsecure.entrypoints=smtpsecure"
      - "traefik.tcp.routers.smtpsecure.rule=HostSNI(`mail.example.com`)"
      - "traefik.tcp.routers.smtpsecure.service=smtp"
      - "traefik.tcp.services.smtp.loadbalancer.server.port=587"
      - "traefik.tcp.services.smtp.loadbalancer.proxyprotocol.version=2"
      - "traefik.tcp.routers.imapsecure.tls=true"
      - "traefik.tcp.routers.imapsecure.tls.certresolver=letsencrypt"
      - "traefik.tcp.routers.imapsecure.entrypoints=imapsecure"
      - "traefik.tcp.routers.imapsecure.rule=HostSNI(`mail.example.com`)"
      - "traefik.tcp.routers.imapsecure.service=imap"
      - "traefik.tcp.services.imap.loadbalancer.server.port=143"
      - "traefik.tcp.services.imap.loadbalancer.proxyprotocol.version=2"
      - "traefik.docker.network=docker_stalwart-internal"
###
    networks:
      - internet
      - stalwart

[jayknyn]

Hi @mdecimus so that Traefik log is showing that only the HTTP request is not being sent with proxy headers. Traefik is sending the proxy headers for the TCP requests (smtp, imap, etc.). As @lostb1t states below, Traefik can't enable ProxyProtocol on HTTP Services. Not sure about other reverse proxies.

I've taken a stab at the code and this is what I think would be needed for stalwart to expect proxy headers on all non-HTTP listeners:
in src/config/server/listerner.rs

        // Parse proxy networks only for TCP listeners
        let mut proxy_networks = Vec::new();
        if protocol != ServerProtocol::Http {
            let proxy_keys = if config.has_prefix(("server.listener", id, "proxy.trusted-networks")) {
                ("server.listener", id, "proxy.trusted-networks").as_key()
            } else {
                "server.proxy.trusted-networks".as_key()
            };
            for (_, network) in config.properties(proxy_keys) {
                proxy_networks.push(network);
            }
        }

Would you consider updating listener.rs to check for proxy headers for all non-HTTP requests if the trusted-networks config is set? Thank you.

[lostb1t]

same issue, it seems stalwart expects proxy protocol to be present for http requests. But the proxy protocol is not supported for http routing in traefik: traefik/traefik#10608

[mdecimus]

@jayknyn @lostb1t You should then disable the global proxy protocol setting and configure it only on the listeners where Traefik is providing these headers.

[lostb1t]

tried the other way around. Kept global used overide proxy on the http listener. Which did not work. which might be considered a bug.

Disabling global and use individual override does indeed resolve the issue