Add logging by default

[jmkhael]

When trying to benchmarking my service with Apache Benchmark, I was faced with the dreaded apr_socket_recv: Connection reset by peer (104)

ab -c 50 -n 100000 http://stratus-clay/demo/hello

I though maybe the problem is HAProxy dropping connections, and I found that there is no default logging active. I'll rebuild the Docker container as per your instructions, but I think it is valuable to have the logs on by default. Or am I missing something? (like I usually do)

(I still haven't found out the initial problem of connections being dropped though :( )

[vfarcic]

Sorry for not responding earlier. I had too much work on my plate...

I started working on logging and realized that it is not as easy as I hoped. HAProxy does provide a mechanism to log to stdout. It uses rsyslog. The problem is that the only acceptable logs output inside Docker containers is stdout and I could not (yet) figure out how to make HAProxy use stdout. I'll give it another try next weekend.

I'm truly sorry if this delay affects you. In the mean time, while I'm trying to figure out how to implement logging to stdout, we can have a chat and share your screen. I'd probably be able to tell you why the proxy drops connections if I have a look at your configs.

[jmkhael]

No worries! This doesn't delay me per se, it delays my learning on this part, that's all :)

I was trying to benchmark the effect of HAProxy caching on my docker swarm services instead of a direct hit on them. The initial investigation of the connection drops seemed to be that the virtual machine suspected being flooded by SYN attack... I couldn't workaround that quickly so i abandoned for now (I've documented it here on my personal blog)

I want to benchmark several reverse proxy solution for Swarm to see which perform better, is easier to setup... I've tried jwilder/nginx but is still doesn't support Swarm mode - this issue should bring that on.

Right now I am playing with Traefik, seems nice, have an ARM docker image for me to play with, but google searches shows that it isn't as fast as HAProxy (Traefik is still very young.)

[vfarcic]

I think that in most cases nginx and HAProxy are unbeatable. That's why I thought to add a level of dynamism on top of HAProxy instead creating a completely new solution.

I'll get back to you once I resolve the logging issue. Until then, please don't hesitate to ask any question you might have about Docker Flow: Proxy.

[vfarcic]

My tests with workarounds that would write to stdout showed a decrease in performance on high loads so I must agree with Willy (HAProxy author) that stdout is not a good option (see this conversation. The alternatives is to use syslog as the only supported logging mechanism with HAProxy.

For a conversation on that subject, please see the HAProxy Issue #3

[vfarcic]

Available in the release 1.248.

[emanuil-tolev]

My tests with workarounds that would write to stdout showed a decrease in performance on high loads so I must agree with Willy (HAProxy author) that stdout is not a good option (see this conversation. The alternatives is to use syslog as the only supported logging mechanism with HAProxy.

The nginx docker image writes to stdout and stderr (/proc/1/fd/1). Is there something we are missing?

[vfarcic]

I did modify DFP to output syslog (the only logging method in HAProxy) to stdout. As for performance, it is true that on very high loads it degrades with outputs to stdout. The fact that nginx does it does not mean that its performance does not degrade because of it. I haven't done those types of tests on nginx (with and without stdout).

Anyways, for better or worse, syslog is the only option. On the other hand, that is not an issue with DFP since I'm capturing syslog entries and sending them to stdout. Unless you have a truly huge load, it should not be a problem. Even then, it is trivial to scale up.

One thing that does really matter is to limit logs to, for example, errors. Do no log every request. That is truly dangerous no matter whether it is nginx, haproxy, or anything else.