worker not restart, (max number of session connection reached, flushing the accept queue)

[5witchkr]

When generating a lot of traffic at once,
"ERROR max number of session connection reached, flushing the accept queue" and the worker dies.
A dead worker will not be restarted.
Assuming there are 4 workers, if a worker dies one by one and eventually all workers die, sozu will not work.

I set the following configurations on Apache JMeter: Number of Threads to 2000, Ramp-up period to 1sec, and Loop Count to Infinite. When I sent about 2 million requests, the workers started to die in chains.

my config.toml

log_level = "trace"
log_target = "stdout"

max_connections = 30000
worker_count = 4
worker_automatic_restart = true

[[listeners]]
protocol = "http"
address = "0.0.0.0:7171"

[clusters]
[clusters.myc]
protocol = "http"

frontends = [ { address = "0.0.0.0:7171", hostname = "myhost" } ]

backends = [ { address = "127.0.0.1:7173" } ]

linux System resource

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 385941
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65536
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 385941
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

error logs

WRK-02 ERROR        max number of session connection reached, flushing the accept queue
WRK-03 ERROR        max number of session connection reached, flushing the accept queue
WRK-01 ERROR        max number of session connection reached, flushing the accept queue
WRK-00 ERROR        max number of session connection reached, flushing the accept queue

[Keksoj]

Thank you for this, I am investigating. It is very probable, as mentionned by @Wonshtrum , that the big HTTP refactoring we did messed with the session manager.

[Keksoj]

@5witchkr it seems you were benchmarking Sōzu, and as far as we guess with @Wonshtrum , it seems Sōzu behaves normally in your, the workers do not die, they refuse new connections and wait for the current ones to go under the 90% capacity limit until accepting new connections.
Can you confirm the workers, in your case, ceased to process current connections?

We did not benchmark Sōzu a lot lately, we would love if you shared your workflow with JMeter in detail, for us to reproduce.

[5witchkr]

@Keksoj Thank you for your response regarding the issue.
I will try to recreate my previous work and provide a detailed workflow with JMeter as soon as possible.
As far as I remember, during my previous attempt, I waited and sent another single request, considering the possibility of a connection capacity limit.
However, I did not receive any response from the browser or curl.
Additionally, when I checked the running processes using the Linux top command, I could not find the Sōzu process.