c2s looping on epoll with "Too many open files"

[crosser]

This only happened once by now, and may be a freak accident, feel free to close if not interested. This morning, I found my clients not able to connect to the server, and when I looked I found /usr/sbin/jabberd2-c2s using 100% CPU, with strace looking like this:

epoll_wait(4, [{EPOLLIN, {u32=4008920096, u64=94304310871072}}], 32, 5000) = 1
accept(6, 0x7ffc3f008ee0, 0x7ffc3f008ecc) = -1 EMFILE (Too many open files)
epoll_wait(4, [{EPOLLIN, {u32=4008920096, u64=94304310871072}}], 32, 5000) = 1
accept(6, 0x7ffc3f008ee0, 0x7ffc3f008ecc) = -1 EMFILE (Too many open files)
epoll_wait(4, [{EPOLLIN, {u32=4008920096, u64=94304310871072}}], 32, 5000) = 1
accept(6, 0x7ffc3f008ee0, 0x7ffc3f008ecc) = -1 EMFILE (Too many open files)
^Cstrace: Process 9259 detached

jabberd2 is from official ubuntu (LTS) package, 2.3.4-1ubuntu2.

[crosser]

Actually this started to happen almost every day.
I will try to collect more information next time it happens.

[crosser]

I am getting this situation pretty regularly these days. Apparently it's denial of service condition when a client opens many connections but does not proceed to login. Perhaps such connections ought to be closed after a short (several seconds) timeout?

strace:

...
accept(6, 0x7ffc1af36d40, 0x7ffc1af36d2c) = -1 EMFILE (Too many open files)
epoll_wait(4, [{EPOLLIN, {u32=3279412352, u64=94187617246336}}], 32, 5000) = 1
accept(6, 0x7ffc1af36d40, 0x7ffc1af36d2c) = -1 EMFILE (Too many open files)
epoll_wait(4, [{EPOLLIN, {u32=3279412352, u64=94187617246336}}], 32, 5000) = 1
accept(6, 0x7ffc1af36d40, 0x7ffc1af36d2c) = -1 EMFILE (Too many open files)
epoll_wait(4, [{EPOLLIN, {u32=3279412352, u64=94187617246336}}], 32, 5000) = 1
...

netstat -atp:

...
9978/jabberd2-c2s
tcp6       0      0 dehost:xmpp-client      36.108.221.62.dyn:54114 ESTABLISHED 9978/jabberd2-c2s
tcp6       0      0 dehost:xmpp-client      36.108.221.62.dyn:62613 ESTABLISHED 9978/jabberd2-c2s
tcp6       0      0 dehost:xmpp-client      36.108.221.62.dyn:61513 ESTABLISHED 9978/jabberd2-c2s
tcp6       0      0 dehost:xmpp-client      36.108.221.62.dyn:49635 ESTABLISHED 9978/jabberd2-c2s
...

syslog:

...
eb 07 07:29:18 dehost jabberd/c2s[9978]: PAM unable to dlopen(pam_umask.so): /lib/security/pam_umask.so: cannot open shared object file: Too many open files
...
Feb 07 07:29:45 dehost jabberd/c2s[9978]: PAM _pam_load_conf_file: unable to open /etc/pam.d/common-auth
Feb 07 07:29:45 dehost jabberd/c2s[9978]: PAM error loading (null)
...
Feb 07 09:31:35 dehost jabberd/c2s[9978]: [9] [::ffff:62.221.108.36, port=52305] disconnect jid=unbound, packets: 0, bytes: 130
Feb 07 09:31:35 dehost jabberd/c2s[9978]: [9] [::ffff:62.221.108.36, port=52314] connect
Feb 07 09:48:06 dehost jabberd/c2s[9978]: [9] [::ffff:62.221.108.36, port=52314] disconnect jid=unbound, packets: 0, bytes: 130
Feb 07 09:48:06 dehost jabberd/c2s[9978]: [9] [::ffff:62.221.108.36, port=52375] connect
...