DIND, Swarm, or Traefik is stripping my X-Forwarded-For header, can we rule out DIND?

[jerrac]

I've posted here: moby/moby#25526 (comment) and here: https://community.traefik.io/t/how-can-i-get-x-forwarded-for-working-its-currently-only-containing-a-single-ip-when-there-should-be-more/14767 already.

Long story short, I should be seeing multiple ip addresses listed in my X-Forwarded-For header by the time it gets to my container. I can rule out my external load balancer because other systems downstream from the LB, XFF has at least 3 ips in it.

By the time XFF hits the httpd container I'm testing with, it only has the LB's floating ip in it. (Before installing DIND, it only had a docker ip in it.)

I think the route my requests take is something like Client -> LB -> Swarm host -> Docker Ingress Network-> Traefik Overlay Network -> Container.

The one part I'm not sure of is if the Ingress network or the Traefik network gets the request first. I think it's the Ingress network since Traefik is running as a globally deployed swarm service.

Any insight would be appreciated.

Thanks in advance!

[jerrac]

Update (posting this in all 3 places I asked for help...):
I turned off the docker-ingress-routing-daemon and configured the ports on my Traefik service to set mode: host. This left me with just the load balancer's ip in XFF. The same effect running the daemon had.

A while later, after a meeting, I randomly decided to try terminating https at my external load balancer. After re-configuring Traefik to not redirect to 443, and configuring my service to use the 80 entrypoint, I can see my client ip in my container logs.

Does this make sense?

[struanb]

To be clear, DIND exists to transform Docker's ingress routing mesh to use policy routing instead of SNAT, to redirect client traffic to service nodes. It will only work to preserve the client IP if incoming requests directly reach a load-balancer node on a port published for a service via the ingress routing mesh. DIND is a network-layer tool (IPv4) and cannot inspect or modify HTTP headers.

I understand Traefik has often been used as a reverse proxy to work around the same limitation as DIND. In this model, incoming requests much directly reach the reverse proxy, which presumably must not be using the ingress routing mesh, but instead have its ports published using host mode, and be launched using --mode global. The Traefik reverse proxy will see the client IP of requests and can add these to the XFF header before reverse proxying them to an internal application service.

DIND therefore exists to solve a similar problem as a Traefik reverse proxy service placed in front of an internal application service, but without the need for an extra service, for proxying, or for introduction/modification of XFF headers, and therefore without modification of the application service (if it doesn't natively support XFF headers).

Combining DIND with Traefik should allow Traefik itself to be deployed using the ingress routing mesh, which could be useful if Traefik is providing additional benefits in one's setup.

However, I'm not sure I can see a use-case for combining DIND with an internal application service published via the ingress routing mesh, and still fronted by a Traefik reverse proxy. Since the reverse proxy node is the client for the internal application service request, doing this will just expose the Docker network IP of that node, instead of the ingress network IP, to the internal application service.

Hope this makes sense.

[jerrac]

Yes, that makes sense. Thanks for the clear explanation. :)

Traefik also provides routing based on hostname. So I think I'll likely stick to using it.

Thanks again, your help was appreciated!