ejabberd frequently crashing on one server

[jtsmartins]

We have two servers running an old ejabberd version (v18), and for the past months one of the servers is frequently crashing.

We've been trying to figure why our ejabberd is crashing, but we may need your expertise reading the log.

By looking at the times and message sequence, I think the first important message is a CRASH REPORT entry with the following exception:

exception error: {{badmatch,ignore},[{xmpp_socket,start,4,[{file,"src/xmpp_socket.erl"},{line,108}]},{ejabberd_listener,accept,4,[{file,"src/ejabberd_listener.erl"},{line,297}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]} ancestors: [ejabberd_listener,ejabberd_sup,<0.66.0>]

After this error we see other entries from the SUPERVISOR about childs being terminated and some about starting them. Sometimes this passes without any problems but, when the ejabberd really crashes, we see shutdwon errors due to reached_max_restart_intensity - you can see this in the attached log.

Both servers have the same configuration, run in VM's with the same specs and are in the same site, so this is strange.
Can someone help me understand what is causing this?

ejabberd_crash.log

[jtsmartins]

Adding the last entries from the ejabberd.log

...
2024-05-01 20:54:36.000 [warning]  lager_file_backend dropped 119 messages in the last second that exceeded the limit of 100 messages/sec
2024-05-01 20:54:36.000 [debug] <0.12161.0>@shaper:update:143 State: {maxrate,1000,0.008335671329974105,1714589555995521}, Size=1
M=0.500002083926518, I=120005.262
2024-05-01 20:54:36.004 [debug] <0.27714.2>@shaper:update:143 State: {maxrate,1000,15.554873741353436,1714589556003074}, Size=1
M=0.5039191997640872, I=120001.251

...
2024-05-01 20:54:36.925 [debug] <0.597.0>@ejabberd_router:unregister_route:208 Route unregistered: irc.cpecs-xmpp
2024-05-01 20:54:36.925 [debug] <0.595.0>@mod_pubsub:terminate_plugins:390 ** terminate flat plugin
2024-05-01 20:54:36.925 [debug] <0.595.0>@mod_pubsub:terminate_plugins:390 ** terminate hometree plugin
2024-05-01 20:54:36.925 [debug] <0.595.0>@mod_pubsub:terminate_plugins:390 ** terminate pep plugin
2024-05-01 20:54:36.926 [debug] <0.595.0>@ejabberd_router:unregister_route:208 Route unregistered: pubsub.cpecs-xmpp
2024-05-01 20:54:36.926 [debug] <0.589.0>@ejabberd_router:unregister_route:208 Route unregistered: echo.cpecs-xmpp
2024-05-01 20:54:36.927 [debug] <0.568.0>@ejabberd_router:unregister_route:208 Route unregistered: vjud.cpecs-xmpp
2024-05-01 20:54:36.957 [debug] <0.559.0>@ejabberd_router:unregister_route:208 Route unregistered: conference.cpecs-xmpp
2024-05-01 20:54:37.244 [debug] <0.541.0>@ejabberd_router:unregister_route:208 Route unregistered: cpecs-xmpp
2024-05-01 20:54:37.245 [debug] <0.474.0>@ejabberd_pkix:clean_dir:618 Cleaning directory /opt/ejabberd/var/lib/ejabberd/certs
2024-05-01 20:54:37.249 [info] <0.66.0>@ejabberd_app:stop:82 ejabberd 0.0 is stopped in the node 'ejabberd@cpecs-xmpp-prod-02'
2024-05-01 20:54:37.249 [error] <0.66.0> application_master: shutdown_error, ejabberd_app: {prep_stop,[[]]}, error_info: {noproc,{gen_server,call,[ejabberd_listener,{terminate_child,{5222,{0,0,0,0,0,0,0,0},tcp}},infinity]}}
2024-05-01 20:54:37.280 [info] <0.33.0> Application ejabberd exited with reason: shutdown

Adding entries from the error.log

...
2024-05-01 20:46:20.223 [error] <0.22795.3>@xmpp_socket:start:108 CRASH REPORT Process <0.22795.3> with 0 neighbours crashed with reason: no match of right hand value ignore in xmpp_socket:start/4 line 108
2024-05-01 20:51:08.547 [error] <0.482.0>@ejabberd_listener:validate_module_option:615 unknown listen option 'use_proxy_protocol' for 'ejabberd_c2s' will be likely ignored, available options are: access, shaper, certfile, ciphers, dhfile, cafile, protocol_options, tls, tls_compression, starttls, starttls_required, tls_verify, zlib, max_fsm_queue
2024-05-01 20:54:35.904 [error] <0.24095.3>@ejabberd_listener:socket_error:480 Failed to open socket:
  {5222,ejabberd_c2s,[inet6,{ip,{0,0,0,0,0,0,0,0}}]}
Reason: IP address and port number already used: :: 5222
2024-05-01 20:54:37.249 [error] <0.66.0> application_master: shutdown_error, ejabberd_app: {prep_stop,[[]]}, error_info: {noproc,{gen_server,call,[ejabberd_listener,{terminate_child,{5222,{0,0,0,0,0,0,0,0},tcp}},infinity]}}

Looking at the first entry in the ejabberd.log above, can this be triggered by an overload?

[prefiks]

What exact version are you using?

[jtsmartins]

How can I have the exact version? I just know that the product uses a ejabberd based in v18.

I'm asking this because when I do a status, I'm getting:

ejabberd 0.0 is running in that node

... what is not compliant to the information shared by our development.

[prefiks]

Looks like you are using 18.01 from those error. Possibly adding supervisor: false to port 5222 listener options will help this. But what would really help is updating to more recent version.

[jtsmartins]

@prefiks, what is the impact of adding the "supervisor: false" in the listener configuration?
(I'm sorry for not doing this sooner, I'm adding the ejabberd.yml.)

We have the update of ejabberd in our scope, we cannot do this in the next months. But that is being considered, since we also see a possible memory leak on this server (once again, only in one of the servers), and we suspect that is related to the version.

ejabberd.yml.txt

[jtsmartins]

@prefiks ,

Today I added the supervisor: false into the /opt/ejabberd/etc/ejabberd/ejabberd.yml and restarted ejabberd (can be seen below). Despite that, I still see new entries in the crash.log from the supervisor for port 5222 (you can also confirm this below).

Is this the correct way of disabling the supervisor?

150 listen:
151   -
152     port: 5222
153     use_proxy_protocol: true
154     ip: "::"
155     module: ejabberd_c2s
156     ##
157     ## If TLS is compiled in and you installed a SSL
158     ## certificate, uncomment this line:
159     ##
160     ## starttls: true
161     ##
162     ## To enforce TLS encryption for client connections,
163     ## use this instead of the "starttls" option:
164     ##
165     ## starttls_required: true
166     ##
167     ## Stream compression
168     ##
169     ## zlib: true
170     ##
171     max_stanza_size: 65536
172     shaper: c2s_shaper
173     access: c2s
174     resume_timeout: 3660
175     supervisor: false
176   -

4287 1 13 15:21 ? 00:24:09 /usr/lib/erlang/erts-9.0/bin/beam.smp -K true -P 25000000 -- -root /usr/lib/erlang -progname erl -- -home /home/axiros -- -sname ejabberd@cpecs-xmpp-prod-02 -smp auto -mnesia dump_log_write_threshold 50000 -mnesia dc_dump_limit 40 -mnesia dir "/opt/ejabberd/var/lib/ejabberd" -ejabberd log_rate_limit 100 log_rotate_size 2684354560 log_rotate_count 1 log_rotate_date "" -s ejabberd -noshell -noinput -noshell -noinput

2024-05-03 18:09:47 =SUPERVISOR REPORT====
     Supervisor: {local,ejabberd_listener}
     Context:    child_terminated
     Reason:     {{badmatch,ignore},[{xmpp_socket,start,4,[{file,"src/xmpp_socket.erl"},{line,108}]},{ejabberd_listener,accept,4,[{file,"src/ejabberd_listener.erl"},{line,297}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}
     Offender:   [{pid,<0.486.0>},{id,{5222,{0,0,0,0,0,0,0,0},tcp}},{mfargs,{ejabberd_listener,start,[{5222,{0,0,0,0,0,0,0,0},tcp},ejabberd_c2s,[{supervisor,false},{resume_timeout,3660},{access,c2s},{shaper,c2s_shaper},{max_stanza_size,65536},{use_proxy_protocol,true}]]}},{restart_type,transient},{shutdown,brutal_kill},{child_type,worker}]

[jtsmartins]

Yes, I see an entry for wrong options in the same listener.

2024-05-06 07:56:21.444 [error] <0.479.0>@ejabberd_listener:validate_module_option:615 unknown listen option 'use_proxy_protocol' for 'ejabberd_c2s' will be likely ignored, available options are: access, shaper, certfile, ciphers, dhfile, cafile, protocol_options, tls, tls_compression, starttls, starttls_required, tls_verify, zlib, max_fsm_queue

These options that are failing are from the listener where we set the supervisor to false. Does it fail to read the other options due to the failure of these ones?

I'm attaching the startup log.

150 listen:
151   -
152     port: 5222
153     use_proxy_protocol: true
154     ip: "::"
155     module: ejabberd_c2s
156     ##
157     ## If TLS is compiled in and you installed a SSL
158     ## certificate, uncomment this line:
159     ##
160     ## starttls: true
161     ##
162     ## To enforce TLS encryption for client connections,
163     ## use this instead of the "starttls" option:
164     ##
165     ## starttls_required: true
166     ##
167     ## Stream compression
168     ##
169     ## zlib: true
170     ##
171     max_stanza_size: 65536
172     shaper: c2s_shaper
173     access: c2s
174     resume_timeout: 3660
175     supervisor: false

ejabberd_startup.txt

[prefiks]

Well that error is understandable, this version doesn't support that option, it was added in later release.

[jtsmartins]

Do you have any idea if we can disable the supervisor in this version?

[prefiks]

Could you remove use_proxy_protocol: true and see if you get error about supervisor option being not supported?

[jtsmartins]

I just removed it and restarted the process.
Similar to the previous restart to apply the change, we see it starting with the supervisor set as false (as seen in the log line below), but lets see if we have new entries of the supervisor in the crash.log after we removed the use_proxy_protocol.

2024-05-06 14:54:13.483 [debug] <0.485.0> Supervisor ejabberd_listener started ejabberd_listener:start({5222,{0,0,0,0,0,0,0,0},tcp}, ejabberd_c2s, [{supervisor,false},{resume_timeout,3660},{access,c2s},{shaper,c2s_shaper},{max_stanza_size,65536}]) at pid <0.486.0>

[prefiks]

So, i don't see other way to fix this that try to update.

[jtsmartins]

I accept this as the final answer.
Thank you for all the support provided.