Jasmin Crashes on High Load

[vaska94]

2023-12-23 16:07:41 INFO 722 Reconnecting after 10 seconds ... 2023-12-23 16:07:44 INFO 735 Connecting to IPv4Address(type='TCP', host='127.0.0.1', port=5672) ... 2023-12-23 16:07:44 INFO 729 Connecting to IPv4Address(type='TCP', host='127.0.0.1', port=5672) ... 2023-12-23 16:07:44 INFO 735 Connection made to 127.0.0.1:5672 2023-12-23 16:07:44 INFO 729 Connection made to 127.0.0.1:5672 2023-12-23 16:07:44 INFO 735 Successfull authentication 2023-12-23 16:07:44 INFO 735 Got channel 2023-12-23 16:07:44 INFO 729 Successfull authentication 2023-12-23 16:07:44 INFO 729 Got channel 2023-12-23 16:07:44 INFO 735 The channel is open 2023-12-23 16:07:44 INFO 729 The channel is open 2023-12-23 16:07:44 ERROR 735 Channel open failed: [Failure instance: Traceback: <class 'twisted.internet.defer.AlreadyCalledError'>: /opt/jasmin-sms-gateway/venv/lib64/python3.9/site-packages/twisted/internet/defer.py:1946:_gotResultInlineCallbacks /opt/jasmin-sms-gateway/venv/lib64/python3.9/site-packages/twisted/internet/defer.py:2085:_inlineCallbacks /opt/jasmin-sms-gateway/venv/lib64/python3.9/site-packages/twisted/internet/defer.py:874:callback /opt/jasmin-sms-gateway/venv/lib64/python3.9/site-packages/twisted/internet/defer.py:981:_startRunCallbacks --- <exception caught here> --- /opt/jasmin-sms-gateway/venv/lib64/python3.9/site-packages/twisted/internet/defer.py:1075:_runCallbacks /opt/jasmin-sms-gateway/venv/lib64/python3.9/site-packages/jasmin/queues/factory.py:181:_channel_open /opt/jasmin-sms-gateway/venv/lib64/python3.9/site-packages/twisted/internet/defer.py:874:callback /opt/jasmin-sms-gateway/venv/lib64/python3.9/site-packages/twisted/internet/defer.py:969:_startRunCallbacks ] 2023-12-23 16:07:44 ERROR 729 Channel open failed: [Failure instance: Traceback: <class 'twisted.internet.defer.AlreadyCalledError'>: /opt/jasmin-sms-gateway/venv/lib64/python3.9/site-packages/twisted/internet/defer.py:1946:_gotResultInlineCallbacks /opt/jasmin-sms-gateway/venv/lib64/python3.9/site-packages/twisted/internet/defer.py:2085:_inlineCallbacks /opt/jasmin-sms-gateway/venv/lib64/python3.9/site-packages/twisted/internet/defer.py:874:callback /opt/jasmin-sms-gateway/venv/lib64/python3.9/site-packages/twisted/internet/defer.py:981:_startRunCallbacks --- <exception caught here> --- /opt/jasmin-sms-gateway/venv/lib64/python3.9/site-packages/twisted/internet/defer.py:1075:_runCallbacks /opt/jasmin-sms-gateway/venv/lib64/python3.9/site-packages/jasmin/queues/factory.py:181:_channel_open /opt/jasmin-sms-gateway/venv/lib64/python3.9/site-packages/twisted/internet/defer.py:874:callback /opt/jasmin-sms-gateway/venv/lib64/python3.9/site-packages/twisted/internet/defer.py:969:_startRunCallbacks ] 2023-12-23 16:07:51 INFO 722 Connecting to IPv4Address(type='TCP', host='127.0.0.1', port=5672) ... 2023-12-23 16:07:51 INFO 722 Connection made to 127.0.0.1:5672 2023-12-23 16:07:51 INFO 722 Successfull authentication 2023-12-23 16:07:51 INFO 722 Got channel 2023-12-23 16:07:51 INFO 722 The channel is open 2023-12-23 16:07:51 ERROR 722 Channel open failed: [Failure instance: Traceback: <class 'twisted.internet.defer.AlreadyCalledError'>: /opt/jasmin-sms-gateway/venv/lib64/python3.9/site-packages/twisted/internet/defer.py:1946:_gotResultInlineCallbacks /opt/jasmin-sms-gateway/venv/lib64/python3.9/site-packages/twisted/internet/defer.py:2085:_inlineCallbacks /opt/jasmin-sms-gateway/venv/lib64/python3.9/site-packages/twisted/internet/defer.py:874:callback /opt/jasmin-sms-gateway/venv/lib64/python3.9/site-packages/twisted/internet/defer.py:981:_startRunCallbacks --- <exception caught here> --- /opt/jasmin-sms-gateway/venv/lib64/python3.9/site-packages/twisted/internet/defer.py:1075:_runCallbacks /opt/jasmin-sms-gateway/venv/lib64/python3.9/site-packages/jasmin/queues/factory.py:181:_channel_open /opt/jasmin-sms-gateway/venv/lib64/python3.9/site-packages/twisted/internet/defer.py:874:callback /opt/jasmin-sms-gateway/venv/lib64/python3.9/site-packages/twisted/internet/defer.py:969:_startRunCallbacks

any ideas?

[ibroheem]

Have you set limits on Linux ?
IF ulimit -n shows a low number like 1024, you might have issues.
edit /etc/security/limits.conf, then add:

* soft nofile 1000000
* hard nofile 1000000

You should also try moving Redis and RabbitMQ to another server.