Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

0.11 AMQP/Rabbit reconnect issue #1162

Open
marcinkowalczyk opened this issue Nov 23, 2023 · 10 comments
Open

0.11 AMQP/Rabbit reconnect issue #1162

marcinkowalczyk opened this issue Nov 23, 2023 · 10 comments

Comments

@marcinkowalczyk
Copy link

marcinkowalczyk commented Nov 23, 2023 •
edited

I think it's issue that is very long in a design. Seems that after disconnect from AMQ Jasmin cannot recover

2023-11-23 13:20:30 INFO     1 Connection lost. Reason: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionDone'>: Connection was closed cleanly.
]
2023-11-23 13:20:30 INFO     1 Reconnecting after 10 seconds ...
2023-11-23 13:20:33 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': '262A8C70', 'headers': {'type': 'deliver_sm', 'cid': 'ALPHA_KAT_SMSC5', 'dlr_dlvrd': '001', 'dlr_sub': '001', 'dlr_sdate': '2311231420', 'dlr_ddate': '2311231420', 'dlr_err': '000', 'dlr_text': '📦 Dziekujemy z', 'dlr_id': '262A8C70', 'dlr_stat': 'DELIVRD'}}>}
2023-11-23 13:20:40 INFO     1 Connecting to IPv4Address(type='TCP', host='rabbit-mq', port=5672) ...
2023-11-23 13:20:40 INFO     1 Connection made to rabbit-mq:5672
2023-11-23 13:20:40 INFO     1 Successfull authentication
2023-11-23 13:20:40 INFO     1 Got channel
2023-11-23 13:20:40 INFO     1 The channel is open
2023-11-23 13:20:40 ERROR    1 Channel open failed: [Failure instance: Traceback: <class 'twisted.internet.defer.AlreadyCalledError'>:
/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py:1946:_gotResultInlineCallbacks
/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py:2085:_inlineCallbacks
/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py:874:callback
/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py:981:_startRunCallbacks
--- <exception caught here> ---
/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py:1075:_runCallbacks
/usr/local/lib/python3.11/site-packages/jasmin/queues/factory.py:181:_channel_open
/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py:874:callback
/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py:969:_startRunCallbacks
]
2023-11-23 13:31:51 ERROR    1 Connection lost. Reason: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion: Connection lost.
]
2023-11-23 13:31:51 INFO     1 Reconnecting after 10 seconds ...
2023-11-23 13:31:53 INFO     1 Connecting to IPv4Address(type='TCP', host='rabbit-mq', port=5672) ...
2023-11-23 13:31:53 INFO     1 Connection made to rabbit-mq:5672
2023-11-23 13:31:53 INFO     1 Successfull authentication
2023-11-23 13:31:53 INFO     1 Got channel
2023-11-23 13:31:53 INFO     1 The channel is open
2023-11-23 13:31:53 INFO     1 A new queue has been successfully declared [RouterPB_deliver_sm_all]

Rabbit and jasmin are on localhost on docker env with following setting:

[amqp-broker]
host=rabbit-mq
port=5672
heartbeat = 1

Jasmin 0.11.0 official docker image, rabbitmq:3-management-alpine

rabbit stacktrace:

2023-11-23 10:52:24.261333+00:00 [info] <0.11315.0> connection <0.11315.0> (172.28.0.5:44756 -> 172.28.0.3:5672): user 'guest' authenticated and granted access to vhost '/'
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0> ** Generic server <0.11324.0> terminating
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0> ** Last message in was {'$gen_cast',
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                            {method,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                {'basic.publish',0,<<"messaging">>,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                    <<"dlr.deliver_sm">>,false,false},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                {content,60,none,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                    <<32,128,0,0,0,216,4,116,121,112,101,83,0,0,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                      0,10,100,101,108,105,118,101,114,95,115,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                      109,3,99,105,100,83,0,0,0,15,65,76,80,72,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                      65,95,87,65,87,95,83,77,83,67,53,9,100,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                      108,114,95,100,108,118,114,100,83,0,0,0,3,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                      48,48,49,7,100,108,114,95,115,117,98,83,0,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                      0,0,3,48,48,49,9,100,108,114,95,115,100,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                      97,116,101,83,0,0,0,10,50,51,49,49,50,51,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                      49,52,50,48,9,100,108,114,95,100,100,97,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                      116,101,83,0,0,0,10,50,51,49,49,50,51,49,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                      52,50,48,7,100,108,114,95,101,114,114,83,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                      0,0,0,3,48,48,48,8,100,108,114,95,116,101,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                      120,116,83,0,0,0,14,240,159,147,166,32,68,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                      122,105,101,107,117,106,101,109,121,32,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                      122,6,100,108,114,95,105,100,83,0,0,0,8,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                      51,55,69,50,57,52,66,70,8,100,108,114,95,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                      115,116,97,116,83,0,0,0,7,68,69,76,73,86,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                      82,68,8,51,55,69,50,57,52,66,70>>,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                    rabbit_framing_amqp_0_9_1,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                    [<<"DELIVRD">>]},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                flow}}
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0> ** When Server state == {ch,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                          {conf,running,rabbit_framing_amqp_0_9_1,1,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                           <0.11315.0>,<0.11322.0>,<0.11315.0>,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                           <<"172.28.0.5:44756 -> 172.28.0.3:5672">>,undefined,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                           {user,<<"guest">>,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                            [administrator],
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                            [{rabbit_auth_backend_internal,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              #Fun<rabbit_auth_backend_internal.3.20918515>}]},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                           <<"/">>,<<"submit.sm.carr1_pl">>,<0.11316.0>,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                           [],none,1,134217728,1800000,#{},1000000000,false},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                          {lstate,<0.11323.0>,false},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                          none,69085,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                          {0,[],[]},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                          {state,#{},erlang},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                          #{<<"DLRLookup-main">> =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {{amqqueue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               {resource,<<"/">>,queue,<<"DLRLookup-main">>},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               false,false,none,[],<0.771.0>,[],[],[],
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               undefined,undefined,[],[],live,0,[],<<"/">>,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               #{user => <<"guest">>},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               rabbit_classic_queue,#{}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {false,0,false,[]}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                            <<"DLRThrower">> =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {{amqqueue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               {resource,<<"/">>,queue,<<"dlr_thrower">>},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               false,false,none,[],<0.787.0>,[],[],[],
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               undefined,undefined,[],[],live,0,[],<<"/">>,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               #{user => <<"guest">>},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               rabbit_classic_queue,#{}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {false,0,false,[]}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                            <<"RouterPB-billrequests">> =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {{amqqueue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               {resource,<<"/">>,queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                <<"RouterPB_bill_request_submit_sm_resp_all">>},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               false,false,none,[],<0.763.0>,[],[],[],
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               undefined,undefined,[],[],live,0,[],<<"/">>,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               #{user => <<"guest">>},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               rabbit_classic_queue,#{}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {false,0,false,[]}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                            <<"RouterPB-delivers">> =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {{amqqueue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               {resource,<<"/">>,queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                <<"RouterPB_deliver_sm_all">>},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               false,false,none,[],<0.754.0>,[],[],[],
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               undefined,undefined,[],[],live,0,[],<<"/">>,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               #{user => <<"guest">>},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               rabbit_classic_queue,#{}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {false,0,false,[]}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                            <<"SMPPClientFactory-ALPHA_KAT_SMSC5">> =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {{amqqueue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               {resource,<<"/">>,queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                <<"submit.sm.ALPHA_KAT_SMSC5">>},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               false,false,none,[],<0.843.0>,[],[],[],
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               undefined,undefined,[],[],live,0,[],<<"/">>,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               #{user => <<"guest">>},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               rabbit_classic_queue,#{}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {false,1,false,[]}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                            <<"SMPPClientFactory-ALPHA_WAW_SMSC5">> =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {{amqqueue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               {resource,<<"/">>,queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                <<"submit.sm.ALPHA_WAW_SMSC5">>},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               false,false,none,[],<0.835.0>,[],[],[],
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               undefined,undefined,[],[],live,0,[],<<"/">>,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               #{user => <<"guest">>},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               rabbit_classic_queue,#{}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {false,1,false,[]}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                            <<"SMPPClientFactory-KAT_SMSC5">> =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {{amqqueue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               {resource,<<"/">>,queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                <<"submit.sm.KAT_SMSC5">>},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               false,false,none,[],<0.803.0>,[],[],[],
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               undefined,undefined,[],[],live,0,[],<<"/">>,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               #{user => <<"guest">>},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               rabbit_classic_queue,#{}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {false,1,false,[]}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                            <<"SMPPClientFactory-SIPPsm">> =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {{amqqueue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               {resource,<<"/">>,queue,<<"submit.sm.SIPPsm">>},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               false,false,none,[],<0.795.0>,[],[],[],
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               undefined,undefined,[],[],live,0,[],<<"/">>,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               #{user => <<"guest">>},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               rabbit_classic_queue,#{}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {false,1,false,[]}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                            <<"SMPPClientFactory-WAW_SMSC5">> =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {{amqqueue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               {resource,<<"/">>,queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                <<"submit.sm.WAW_SMSC5">>},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               false,false,none,[],<0.811.0>,[],[],[],
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               undefined,undefined,[],[],live,0,[],<<"/">>,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               #{user => <<"guest">>},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               rabbit_classic_queue,#{}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {false,1,false,[]}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                            <<"SMPPClientFactory-WAW_SMSC6">> =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {{amqqueue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               {resource,<<"/">>,queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                <<"submit.sm.WAW_SMSC6">>},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               false,false,none,[],<0.819.0>,[],[],[],
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               undefined,undefined,[],[],live,0,[],<<"/">>,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               #{user => <<"guest">>},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               rabbit_classic_queue,#{}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {false,1,false,[]}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                            <<"SMPPClientFactory-WAW_SMSC7">> =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {{amqqueue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               {resource,<<"/">>,queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                <<"submit.sm.WAW_SMSC7">>},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               false,false,none,[],<0.827.0>,[],[],[],
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               undefined,undefined,[],[],live,0,[],<<"/">>,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               #{user => <<"guest">>},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               rabbit_classic_queue,#{}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {false,1,false,[]}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                            <<"SMPPClientFactory-carr1_pl">> =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {{amqqueue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               {resource,<<"/">>,queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                <<"submit.sm.carr1_pl">>},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               false,false,none,[],<0.1777.0>,[],[],[],
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               undefined,undefined,[],[],live,0,[],<<"/">>,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               #{user => <<"guest">>},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               rabbit_classic_queue,#{}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {false,1,false,[]}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                            <<"SMPPClientFactory-carr1_premium">> =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {{amqqueue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               {resource,<<"/">>,queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                <<"submit.sm.carr1_premium">>},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               false,false,none,[],<0.1755.0>,[],[],[],
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               undefined,undefined,[],[],live,0,[],<<"/">>,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               #{user => <<"guest">>},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               rabbit_classic_queue,#{}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {false,1,false,[]}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                            <<"deliverSmThrower">> =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {{amqqueue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               {resource,<<"/">>,queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                <<"deliver_sm_thrower">>},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               false,false,none,[],<0.777.0>,[],[],[],
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               undefined,undefined,[],[],live,0,[],<<"/">>,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               #{user => <<"guest">>},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               rabbit_classic_queue,#{}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {false,0,false,[]}}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                          #{{resource,<<"/">>,queue,<<"DLRLookup-main">>} =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {1,{<<"DLRLookup-main">>,nil,nil}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                            {resource,<<"/">>,queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             <<"RouterPB_bill_request_submit_sm_resp_all">>} =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {1,{<<"RouterPB-billrequests">>,nil,nil}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                            {resource,<<"/">>,queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             <<"RouterPB_deliver_sm_all">>} =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {1,{<<"RouterPB-delivers">>,nil,nil}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                            {resource,<<"/">>,queue,<<"deliver_sm_thrower">>} =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {1,{<<"deliverSmThrower">>,nil,nil}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                            {resource,<<"/">>,queue,<<"dlr_thrower">>} =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {1,{<<"DLRThrower">>,nil,nil}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                            {resource,<<"/">>,queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             <<"submit.sm.ALPHA_KAT_SMSC5">>} =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {1,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {<<"SMPPClientFactory-ALPHA_KAT_SMSC5">>,nil,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               nil}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                            {resource,<<"/">>,queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             <<"submit.sm.ALPHA_WAW_SMSC5">>} =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {1,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {<<"SMPPClientFactory-ALPHA_WAW_SMSC5">>,nil,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               nil}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                            {resource,<<"/">>,queue,<<"submit.sm.KAT_SMSC5">>} =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {1,{<<"SMPPClientFactory-KAT_SMSC5">>,nil,nil}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                            {resource,<<"/">>,queue,<<"submit.sm.SIPPsm">>} =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {1,{<<"SMPPClientFactory-SIPPsm">>,nil,nil}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                            {resource,<<"/">>,queue,<<"submit.sm.WAW_SMSC5">>} =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {1,{<<"SMPPClientFactory-WAW_SMSC5">>,nil,nil}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                            {resource,<<"/">>,queue,<<"submit.sm.WAW_SMSC6">>} =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {1,{<<"SMPPClientFactory-WAW_SMSC6">>,nil,nil}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                            {resource,<<"/">>,queue,<<"submit.sm.WAW_SMSC7">>} =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {1,{<<"SMPPClientFactory-WAW_SMSC7">>,nil,nil}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                            {resource,<<"/">>,queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             <<"submit.sm.carr1_pl">>} =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {1,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {<<"SMPPClientFactory-carr1_pl">>,nil,nil}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                            {resource,<<"/">>,queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             <<"submit.sm.carr1_premium">>} =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {1,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {<<"SMPPClientFactory-carr1_premium">>,nil,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               nil}}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                          {state,fine,5000,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                           #Ref<0.4056907990.2522349569.125939>},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                          false,1,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                          {rabbit_confirms,undefined,#{}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                          [],[],none,flow,[],
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                          {rabbit_queue_type,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                           #{{resource,<<"/">>,queue,<<"DLRLookup-main">>} =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {ctx,rabbit_classic_queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               {rabbit_classic_queue,<0.771.0>,#{},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                #{<0.771.0> => ok}}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {resource,<<"/">>,queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              <<"RouterPB_bill_request_submit_sm_resp_all">>} =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {ctx,rabbit_classic_queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               {rabbit_classic_queue,<0.763.0>,#{},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                #{<0.763.0> => ok}}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {resource,<<"/">>,queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              <<"RouterPB_deliver_sm_all">>} =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {ctx,rabbit_classic_queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               {rabbit_classic_queue,<0.754.0>,#{},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                #{<0.754.0> => ok}}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {resource,<<"/">>,queue,<<"deliver_sm_thrower">>} =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {ctx,rabbit_classic_queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               {rabbit_classic_queue,<0.777.0>,#{},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                #{<0.777.0> => ok}}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {resource,<<"/">>,queue,<<"dlr_thrower">>} =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {ctx,rabbit_classic_queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               {rabbit_classic_queue,<0.787.0>,#{},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                #{<0.787.0> => ok}}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {resource,<<"/">>,queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              <<"submit.sm.ALPHA_KAT_SMSC5">>} =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {ctx,rabbit_classic_queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               {rabbit_classic_queue,<0.843.0>,#{},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                #{<0.843.0> => ok}}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {resource,<<"/">>,queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              <<"submit.sm.ALPHA_WAW_SMSC5">>} =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {ctx,rabbit_classic_queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               {rabbit_classic_queue,<0.835.0>,#{},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                #{<0.835.0> => ok}}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {resource,<<"/">>,queue,<<"submit.sm.KAT_SMSC5">>} =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {ctx,rabbit_classic_queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               {rabbit_classic_queue,<0.803.0>,#{},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                #{<0.803.0> => ok}}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {resource,<<"/">>,queue,<<"submit.sm.SIPPsm">>} =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {ctx,rabbit_classic_queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               {rabbit_classic_queue,<0.795.0>,#{},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                #{<0.795.0> => ok}}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {resource,<<"/">>,queue,<<"submit.sm.WAW_SMSC5">>} =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {ctx,rabbit_classic_queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               {rabbit_classic_queue,<0.811.0>,#{},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                #{<0.811.0> => ok}}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {resource,<<"/">>,queue,<<"submit.sm.WAW_SMSC6">>} =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {ctx,rabbit_classic_queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               {rabbit_classic_queue,<0.819.0>,#{},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                #{<0.819.0> => ok}}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {resource,<<"/">>,queue,<<"submit.sm.WAW_SMSC7">>} =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {ctx,rabbit_classic_queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               {rabbit_classic_queue,<0.827.0>,#{},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                #{<0.827.0> => ok}}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {resource,<<"/">>,queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              <<"submit.sm.carr1_pl">>} =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {ctx,rabbit_classic_queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               {rabbit_classic_queue,<0.1777.0>,#{},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                #{<0.1777.0> => ok}}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                             {resource,<<"/">>,queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              <<"submit.sm.carr1_premium">>} =>
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                              {ctx,rabbit_classic_queue,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                               {rabbit_classic_queue,<0.1755.0>,#{},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                                #{<0.1755.0> => ok}}}}},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                          #Ref<0.4056907990.2522349569.125937>,true}
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0> ** Reason for termination ==
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0> ** {function_clause,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>        [{rabbit_binary_parser,parse_table,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>             [<<121,32,122,6,100,108,114,95,105,100,83,0,0,0,8,51,55,69,50,57,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                52,66,70,8,100,108,114,95,115,116,97,116,83,0,0,0,7,68,69,76,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>                73,86,82,68>>],
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>             [{file,"rabbit_binary_parser.erl"},{line,41}]},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>         {rabbit_binary_parser,parse_table,1,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>             [{file,"rabbit_binary_parser.erl"},{line,43}]},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>         {rabbit_framing_amqp_0_9_1,decode_properties,2,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>             [{file,"rabbit_framing_amqp_0_9_1.erl"},{line,924}]},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>         {rabbit_binary_parser,ensure_content_decoded,1,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>             [{file,"rabbit_binary_parser.erl"},{line,146}]},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>         {rabbit_channel,handle_method,3,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>             [{file,"rabbit_channel.erl"},{line,1260}]},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>         {rabbit_channel,handle_cast,2,
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>             [{file,"rabbit_channel.erl"},{line,633}]},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>         {gen_server2,handle_msg,2,[{file,"gen_server2.erl"},{line,1056}]},
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>         {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,250}]}]}
2023-11-23 13:20:30.030877+00:00 [error] <0.11324.0>
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>   crasher:
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>     initial call: rabbit_channel:init/1
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>     pid: <0.11324.0>
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>     registered_name: []
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>     exception exit: {function_clause,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                         [{rabbit_binary_parser,parse_table,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                              [<<121,32,122,6,100,108,114,95,105,100,83,0,0,0,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                                 8,51,55,69,50,57,52,66,70,8,100,108,114,95,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                                 115,116,97,116,83,0,0,0,7,68,69,76,73,86,82,68>>],
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                              [{file,"rabbit_binary_parser.erl"},{line,41}]},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                          {rabbit_binary_parser,parse_table,1,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                              [{file,"rabbit_binary_parser.erl"},{line,43}]},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                          {rabbit_framing_amqp_0_9_1,decode_properties,2,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                              [{file,"rabbit_framing_amqp_0_9_1.erl"},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                               {line,924}]},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                          {rabbit_binary_parser,ensure_content_decoded,1,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                              [{file,"rabbit_binary_parser.erl"},{line,146}]},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                          {rabbit_channel,handle_method,3,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                              [{file,"rabbit_channel.erl"},{line,1260}]},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                          {rabbit_channel,handle_cast,2,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                              [{file,"rabbit_channel.erl"},{line,633}]},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                          {gen_server2,handle_msg,2,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                              [{file,"gen_server2.erl"},{line,1056}]},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                          {proc_lib,wake_up,3,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                              [{file,"proc_lib.erl"},{line,250}]}]}
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>       in function  gen_server2:terminate/3 (gen_server2.erl, line 1172)
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>     ancestors: [<0.11321.0>,<0.11319.0>,<0.11314.0>,<0.11313.0>,<0.720.0>,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   <0.719.0>,<0.718.0>,<0.716.0>,<0.715.0>,rabbit_sup,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   <0.230.0>]
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>     message_queue_len: 0
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>     messages: []
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>     links: [<0.11321.0>]
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>     dictionary: [{{queue_exchange_stats,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                        {{resource,<<"/">>,queue,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                             <<"submit.sm.ALPHA_WAW_SMSC5">>},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                         {resource,<<"/">>,exchange,<<"messaging">>}}},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                    none},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {{queue_exchange_stats,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                        {{resource,<<"/">>,queue,<<"submit.sm.WAW_SMSC5">>},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                         {resource,<<"/">>,exchange,<<"messaging">>}}},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                    none},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {{credit_from,<0.835.0>},278},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {{credit_to,<0.11315.0>},123},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {{queue_stats,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                        {resource,<<"/">>,queue,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                            <<"submit.sm.ALPHA_WAW_SMSC5">>}},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                    none},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {{queue_stats,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                        {resource,<<"/">>,queue,<<"submit.sm.WAW_SMSC5">>}},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                    none},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {{xtype_to_module,topic},rabbit_exchange_type_topic},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {{queue_stats,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                        {resource,<<"/">>,queue,<<"submit.sm.SIPPsm">>}},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                    none},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {permission_cache_can_expire,false},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {{credit_from,<0.771.0>},261},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {{credit_from,<0.803.0>},340},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {{queue_exchange_stats,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                        {{resource,<<"/">>,queue,<<"submit.sm.SIPPsm">>},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                         {resource,<<"/">>,exchange,<<"messaging">>}}},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                    none},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {{exchange_stats,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                        {resource,<<"/">>,exchange,<<"messaging">>}},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                    none},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {{queue_stats,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                        {resource,<<"/">>,queue,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                            <<"submit.sm.ALPHA_KAT_SMSC5">>}},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                    none},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {{credit_from,<0.795.0>},305},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {topic_permission_cache,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                       [{{resource,<<"/">>,topic,<<"messaging">>},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                         #{routing_key => <<"dlr.deliver_sm">>,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                           variable_map =>
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                               #{<<"username">> => <<"guest">>,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                                 <<"vhost">> => <<"/">>}},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                         write},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                        {{resource,<<"/">>,topic,<<"messaging">>},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                         #{routing_key => <<"submit.sm.resp.ALPHA_KAT_SMSC5">>,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                           variable_map =>
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                               #{<<"username">> => <<"guest">>,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                                 <<"vhost">> => <<"/">>}},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                         write},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                        {{resource,<<"/">>,topic,<<"messaging">>},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                         #{routing_key => <<"submit.sm.ALPHA_KAT_SMSC5">>,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                           variable_map =>
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                               #{<<"username">> => <<"guest">>,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                                 <<"vhost">> => <<"/">>}},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                         write},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                        {{resource,<<"/">>,topic,<<"messaging">>},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                         #{routing_key => <<"submit.sm.resp.ALPHA_WAW_SMSC5">>,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                           variable_map =>
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                               #{<<"username">> => <<"guest">>,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                                 <<"vhost">> => <<"/">>}},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                         write},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                        {{resource,<<"/">>,topic,<<"messaging">>},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                         #{routing_key => <<"submit.sm.ALPHA_WAW_SMSC5">>,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                           variable_map =>
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                               #{<<"username">> => <<"guest">>,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                                 <<"vhost">> => <<"/">>}},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                         write},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                        {{resource,<<"/">>,topic,<<"messaging">>},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                         #{routing_key => <<"submit.sm.resp.SIPPsm">>,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                           variable_map =>
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                               #{<<"username">> => <<"guest">>,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                                 <<"vhost">> => <<"/">>}},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                         write},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                        {{resource,<<"/">>,topic,<<"messaging">>},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                         #{routing_key => <<"dlr.submit_sm_resp">>,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                           variable_map =>
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                               #{<<"username">> => <<"guest">>,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                                 <<"vhost">> => <<"/">>}},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                         write},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                        {{resource,<<"/">>,topic,<<"messaging">>},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                         #{routing_key => <<"submit.sm.SIPPsm">>,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                           variable_map =>
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                               #{<<"username">> => <<"guest">>,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                                 <<"vhost">> => <<"/">>}},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                         write}]},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {{credit_from,<0.787.0>},220},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {guid,{{938356519,2205611762,2221627935,2567888837},67768}},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {{credit_from,<0.811.0>},343},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {permission_cache,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                       [{{resource,<<"/">>,exchange,<<"messaging">>},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                         #{},write}]},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {{queue_stats,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                        {resource,<<"/">>,queue,<<"submit.sm.KAT_SMSC5">>}},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                    none},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {rand_seed,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                       {#{jump => #Fun<rand.3.34006561>,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                          max => 288230376151711743,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                          next => #Fun<rand.5.34006561>,type => exsplus},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                        [37773391100002717|34174783047712300]}},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {credit_flow_default_credit,{400,200}},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {channel_operation_timeout,15000},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {{queue_stats,{resource,<<"/">>,queue,<<"dlr_thrower">>}},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                    none},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {{credit_from,<0.843.0>},284},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {process_name,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                       {rabbit_channel,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                           {<<"172.28.0.5:44756 -> 172.28.0.3:5672">>,1}}},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {{queue_exchange_stats,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                        {{resource,<<"/">>,queue,<<"submit.sm.KAT_SMSC5">>},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                         {resource,<<"/">>,exchange,<<"messaging">>}}},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                    none},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {{queue_stats,{resource,<<"/">>,queue,<<"DLRLookup-main">>}},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                    none},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {{queue_exchange_stats,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                        {{resource,<<"/">>,queue,<<"dlr_thrower">>},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                         {resource,<<"/">>,exchange,<<"messaging">>}}},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                    none},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {{queue_exchange_stats,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                        {{resource,<<"/">>,queue,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                             <<"submit.sm.ALPHA_KAT_SMSC5">>},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                         {resource,<<"/">>,exchange,<<"messaging">>}}},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                    none},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {{queue_exchange_stats,
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                        {{resource,<<"/">>,queue,<<"DLRLookup-main">>},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                         {resource,<<"/">>,exchange,<<"messaging">>}}},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                    none},
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>                   {msg_size_for_gc,57709516}]
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>     trap_exit: true
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>     status: running
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>     heap_size: 75113
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>     stack_size: 28
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>     reductions: 71969402
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>   neighbours:
2023-11-23 13:20:30.037361+00:00 [error] <0.11324.0>
2023-11-23 13:20:30.044786+00:00 [error] <0.11315.0> Error on AMQP connection <0.11315.0> (172.28.0.5:44756 -> 172.28.0.3:5672, vhost: '/', user: 'guest', state: running), channel 1:
2023-11-23 13:20:30.044786+00:00 [error] <0.11315.0>  {function_clause,
2023-11-23 13:20:30.044786+00:00 [error] <0.11315.0>     [{rabbit_binary_parser,parse_table,
2023-11-23 13:20:30.044786+00:00 [error] <0.11315.0>          [<<121,32,122,6,100,108,114,95,105,100,83,0,0,0,8,51,55,69,50,57,52,
2023-11-23 13:20:30.044786+00:00 [error] <0.11315.0>             66,70,8,100,108,114,95,115,116,97,116,83,0,0,0,7,68,69,76,73,86,
2023-11-23 13:20:30.044786+00:00 [error] <0.11315.0>             82,68>>],
2023-11-23 13:20:30.044786+00:00 [error] <0.11315.0>          [{file,"rabbit_binary_parser.erl"},{line,41}]},
2023-11-23 13:20:30.044786+00:00 [error] <0.11315.0>      {rabbit_binary_parser,parse_table,1,
2023-11-23 13:20:30.044786+00:00 [error] <0.11315.0>          [{file,"rabbit_binary_parser.erl"},{line,43}]},
2023-11-23 13:20:30.044786+00:00 [error] <0.11315.0>      {rabbit_framing_amqp_0_9_1,decode_properties,2,
2023-11-23 13:20:30.044786+00:00 [error] <0.11315.0>          [{file,"rabbit_framing_amqp_0_9_1.erl"},{line,924}]},
2023-11-23 13:20:30.044786+00:00 [error] <0.11315.0>      {rabbit_binary_parser,ensure_content_decoded,1,
2023-11-23 13:20:30.044786+00:00 [error] <0.11315.0>          [{file,"rabbit_binary_parser.erl"},{line,146}]},
2023-11-23 13:20:30.044786+00:00 [error] <0.11315.0>      {rabbit_channel,handle_method,3,
2023-11-23 13:20:30.044786+00:00 [error] <0.11315.0>          [{file,"rabbit_channel.erl"},{line,1260}]},
2023-11-23 13:20:30.044786+00:00 [error] <0.11315.0>      {rabbit_channel,handle_cast,2,[{file,"rabbit_channel.erl"},{line,633}]},
2023-11-23 13:20:30.044786+00:00 [error] <0.11315.0>      {gen_server2,handle_msg,2,[{file,"gen_server2.erl"},{line,1056}]},
2023-11-23 13:20:30.044786+00:00 [error] <0.11315.0>      {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,250}]}]}
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>     supervisor: {<0.11321.0>,rabbit_channel_sup}
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>     errorContext: child_terminated
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>     reason: {function_clause,
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                 [{rabbit_binary_parser,parse_table,
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                      [<<121,32,122,6,100,108,114,95,105,100,83,0,0,0,8,51,55,
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                         69,50,57,52,66,70,8,100,108,114,95,115,116,97,116,83,
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                         0,0,0,7,68,69,76,73,86,82,68>>],
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                      [{file,"rabbit_binary_parser.erl"},{line,41}]},
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                  {rabbit_binary_parser,parse_table,1,
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                      [{file,"rabbit_binary_parser.erl"},{line,43}]},
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                  {rabbit_framing_amqp_0_9_1,decode_properties,2,
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                      [{file,"rabbit_framing_amqp_0_9_1.erl"},{line,924}]},
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                  {rabbit_binary_parser,ensure_content_decoded,1,
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                      [{file,"rabbit_binary_parser.erl"},{line,146}]},
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                  {rabbit_channel,handle_method,3,
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                      [{file,"rabbit_channel.erl"},{line,1260}]},
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                  {rabbit_channel,handle_cast,2,
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                      [{file,"rabbit_channel.erl"},{line,633}]},
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                  {gen_server2,handle_msg,2,
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                      [{file,"gen_server2.erl"},{line,1056}]},
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                  {proc_lib,wake_up,3,[{file,"proc_lib.erl"},{line,250}]}]}
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>     offender: [{pid,<0.11324.0>},
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                {id,channel},
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                {mfargs,
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                    {rabbit_channel,start_link,
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                        [1,<0.11315.0>,<0.11322.0>,<0.11315.0>,
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                         <<"172.28.0.5:44756 -> 172.28.0.3:5672">>,
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                         rabbit_framing_amqp_0_9_1,
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                         {user,<<"guest">>,
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                             [administrator],
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                             [{rabbit_auth_backend_internal,
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                                  #Fun<rabbit_auth_backend_internal.3.20918515>}]},
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                         <<"/">>,[],<0.11316.0>,<0.11323.0>]}},
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                {restart_type,transient},
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                {significant,true},
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                {shutdown,70000},
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>                {child_type,worker}]
2023-11-23 13:20:30.044473+00:00 [error] <0.11321.0>
2023-11-23 13:20:30.096403+00:00 [warning] <0.11315.0> Non-AMQP exit reason '{function_clause,
2023-11-23 13:20:30.096403+00:00 [warning] <0.11315.0>                           [{rabbit_binary_parser,parse_table,
2023-11-23 13:20:30.096403+00:00 [warning] <0.11315.0>                                [<<121,32,122,6,100,108,114,95,105,100,83,0,0,
2023-11-23 13:20:30.096403+00:00 [warning] <0.11315.0>                                   0,8,51,55,69,50,57,52,66,70,8,100,108,114,
2023-11-23 13:20:30.096403+00:00 [warning] <0.11315.0>                                   95,115,116,97,116,83,0,0,0,7,68,69,76,73,86,
2023-11-23 13:20:30.096403+00:00 [warning] <0.11315.0>                                   82,68>>],
2023-11-23 13:20:30.096403+00:00 [warning] <0.11315.0>                                [{file,"rabbit_binary_parser.erl"},{line,41}]},
2023-11-23 13:20:30.096403+00:00 [warning] <0.11315.0>                            {rabbit_binary_parser,parse_table,1,
2023-11-23 13:20:30.096403+00:00 [warning] <0.11315.0>                                [{file,"rabbit_binary_parser.erl"},{line,43}]},
2023-11-23 13:20:30.096403+00:00 [warning] <0.11315.0>                            {rabbit_framing_amqp_0_9_1,decode_properties,2,
2023-11-23 13:20:30.096403+00:00 [warning] <0.11315.0>                                [{file,"rabbit_framing_amqp_0_9_1.erl"},
2023-11-23 13:20:30.096403+00:00 [warning] <0.11315.0>                                 {line,924}]},
2023-11-23 13:20:30.096403+00:00 [warning] <0.11315.0>                            {rabbit_binary_parser,ensure_content_decoded,1,
2023-11-23 13:20:30.096403+00:00 [warning] <0.11315.0>                                [{file,"rabbit_binary_parser.erl"},{line,146}]},
2023-11-23 13:20:30.096403+00:00 [warning] <0.11315.0>                            {rabbit_channel,handle_method,3,
2023-11-23 13:20:30.096403+00:00 [warning] <0.11315.0>                                [{file,"rabbit_channel.erl"},{line,1260}]},
2023-11-23 13:20:30.096403+00:00 [warning] <0.11315.0>                            {rabbit_channel,handle_cast,2,
2023-11-23 13:20:30.096403+00:00 [warning] <0.11315.0>                                [{file,"rabbit_channel.erl"},{line,633}]},
2023-11-23 13:20:30.096403+00:00 [warning] <0.11315.0>                            {gen_server2,handle_msg,2,
2023-11-23 13:20:30.096403+00:00 [warning] <0.11315.0>                                [{file,"gen_server2.erl"},{line,1056}]},
2023-11-23 13:20:30.096403+00:00 [warning] <0.11315.0>                            {proc_lib,wake_up,3,
2023-11-23 13:20:30.096403+00:00 [warning] <0.11315.0>                                [{file,"proc_lib.erl"},{line,250}]}]}'
2023-11-23 13:20:30.096745+00:00 [error] <0.11321.0>     supervisor: {<0.11321.0>,rabbit_channel_sup}
2023-11-23 13:20:30.096745+00:00 [error] <0.11321.0>     errorContext: shutdown
2023-11-23 13:20:30.096745+00:00 [error] <0.11321.0>     reason: reached_max_restart_intensity
2023-11-23 13:20:30.096745+00:00 [error] <0.11321.0>     offender: [{pid,<0.11324.0>},
2023-11-23 13:20:30.096745+00:00 [error] <0.11321.0>                {id,channel},
2023-11-23 13:20:30.096745+00:00 [error] <0.11321.0>                {mfargs,
2023-11-23 13:20:30.096745+00:00 [error] <0.11321.0>                    {rabbit_channel,start_link,
2023-11-23 13:20:30.096745+00:00 [error] <0.11321.0>                        [1,<0.11315.0>,<0.11322.0>,<0.11315.0>,
2023-11-23 13:20:30.096745+00:00 [error] <0.11321.0>                         <<"172.28.0.5:44756 -> 172.28.0.3:5672">>,
2023-11-23 13:20:30.096745+00:00 [error] <0.11321.0>                         rabbit_framing_amqp_0_9_1,
2023-11-23 13:20:30.096745+00:00 [error] <0.11321.0>                         {user,<<"guest">>,
2023-11-23 13:20:30.096745+00:00 [error] <0.11321.0>                             [administrator],
2023-11-23 13:20:30.096745+00:00 [error] <0.11321.0>                             [{rabbit_auth_backend_internal,
2023-11-23 13:20:30.096745+00:00 [error] <0.11321.0>                                  #Fun<rabbit_auth_backend_internal.3.20918515>}]},
2023-11-23 13:20:30.096745+00:00 [error] <0.11321.0>                         <<"/">>,[],<0.11316.0>,<0.11323.0>]}},
2023-11-23 13:20:30.096745+00:00 [error] <0.11321.0>                {restart_type,transient},
2023-11-23 13:20:30.096745+00:00 [error] <0.11321.0>                {significant,true},
2023-11-23 13:20:30.096745+00:00 [error] <0.11321.0>                {shutdown,70000},
2023-11-23 13:20:30.096745+00:00 [error] <0.11321.0>                {child_type,worker}]
2023-11-23 13:20:30.096745+00:00 [error] <0.11321.0>
@farirat
Copy link
Contributor

farirat commented Nov 23, 2023

Does jasmin retry connection when rabbit goes down ?

@marcinkowalczyk
Copy link
Author

Hi,

No jasmin restart is required. It receives messages via smpp, publishes to rabbit but it does not consume until restart.
Rabbit did not go down, it was up all the time.

@farirat
Copy link
Contributor

farirat commented Nov 23, 2023

Do you have any idea of why the consumer stopped in jasmin ?

@marcinkowalczyk
Copy link
Author

Hi,
I don't know reason why it failed - but this is an tcp connection that can always fail. AMQP client should be able to recover. At the moment it stops with

2023-11-23 13:20:40 ERROR    1 Channel open failed: [Failure instance: Traceback: <class 'twisted.internet.defer.AlreadyCalledError'>:

and will not consume new messages until restart.
Same time jasmind is receving smpp messages and publishing them to rabbit queue, just not consuming messages.

@aivisol
Copy link

aivisol commented Nov 30, 2023

Can confirm, I am seeing that as well. Jasmin will not reconnect unless restarted. Happens pretty frequently, I have tried to use various versions of RabbitMQ, it does not help.
My observation is that a specific type of SMPP message causes RabbitMQ to close the connections, but so far I was unable to reproduce it. I think that better re-connection handling on a Jasmin site would definitely help.

The amqp client log is pretty the same a for OP:

2023-11-29 21:58:05 INFO     377878 Connection lost. Reason: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionDone'>: Connection was closed cleanly.
]
2023-11-29 21:58:05 INFO     377878 Reconnecting after 10 seconds ...
2023-11-29 21:58:06 ERROR    377878 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'messag
e-id': '307CA31F', 'headers': {'type': 'deliver_sm', 'cid': 'id9', 'dlr_dlvrd': '001', 'dlr_sub': '001', 'dlr_sdate': '2311291658', 'dlr_ddate': '2311291658', 'dlr_err': '000', 'dlr_text': 'xxxxxxxxxxxxx', 'd
lr_id': '0813474591', 'dlr_stat': 'DELIVRD'}}>}
2023-11-29 21:58:15 INFO     377878 Connecting to IPv4Address(type='TCP', host='localhost', port=5672) ...
2023-11-29 21:58:15 INFO     377878 Connection made to localhost:5672
2023-11-29 21:58:15 INFO     377878 Successfull authentication
2023-11-29 21:58:15 INFO     377878 Got channel
2023-11-29 21:58:15 INFO     377878 The channel is open
2023-11-29 21:58:15 ERROR    377878 Channel open failed: [Failure instance: Traceback: <class 'twisted.internet.defer.AlreadyCalledError'>:

@aivisol
Copy link

aivisol commented Nov 30, 2023

Also to add to OP: RabbitMQ does not go down itself, it just closes that specific connection and expects Jasmin to reconnect.

@vaska94
Copy link

vaska94 commented Dec 2, 2023

Same happens to me once a day, any ideas?

]
2023-11-27 09:16:17 INFO     1 Reconnecting after 10 seconds ...
2023-11-27 09:16:17 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='EXPIRED', children=[], properties={'message-id': 'SM000000030135747955', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000030135747955', 'dlr_stat': 'EXPIRED'}}>}
2023-11-27 09:16:18 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000030141213244', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000030141213244', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:18 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': '4707C68', 'headers': {'type': 'deliver_sm', 'cid': 'cellfie', 'dlr_dlvrd': '001', 'dlr_sub': '001', 'dlr_sdate': '2311271316', 'dlr_ddate': '2311271316', 'dlr_err': '000', 'dlr_text': '\x10\x00 =ܪ\x00', 'dlr_id': '04707C68', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:18 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000029186752066', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000029186752066', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:18 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': '7306068265456243068', 'headers': {'type': 'deliver_sm', 'cid': 'silknet', 'dlr_dlvrd': '001', 'dlr_sub': '001', 'dlr_sdate': '2311271316', 'dlr_ddate': '2311271316', 'dlr_err': '000', 'dlr_text': '', 'dlr_id': '7306068265456243068', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:18 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': '7306068265453031804', 'headers': {'type': 'deliver_sm', 'cid': 'silknet', 'dlr_dlvrd': '001', 'dlr_sub': '001', 'dlr_sdate': '2311271316', 'dlr_ddate': '2311271316', 'dlr_err': '000', 'dlr_text': '', 'dlr_id': '7306068265453031804', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:18 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000030141213251', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000030141213251', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:18 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000030141213264', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000030141213264', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:18 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': '7306068265453293948', 'headers': {'type': 'deliver_sm', 'cid': 'silknet', 'dlr_dlvrd': '001', 'dlr_sub': '001', 'dlr_sdate': '2311271316', 'dlr_ddate': '2311271316', 'dlr_err': '000', 'dlr_text': '', 'dlr_id': '7306068265453293948', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:18 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000030141213274', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000030141213274', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:18 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000030141213229', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000030141213229', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:18 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000030141213246', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000030141213246', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:18 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': '7306068265453162876', 'headers': {'type': 'deliver_sm', 'cid': 'silknet', 'dlr_dlvrd': '001', 'dlr_sub': '001', 'dlr_sdate': '2311271316', 'dlr_ddate': '2311271316', 'dlr_err': '000', 'dlr_text': '', 'dlr_id': '7306068265453162876', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:18 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000030141213270', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000030141213270', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:18 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000030141213266', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000030141213266', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:18 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': '7306068265450541436', 'headers': {'type': 'deliver_sm', 'cid': 'silknet', 'dlr_dlvrd': '001', 'dlr_sub': '001', 'dlr_sdate': '2311271316', 'dlr_ddate': '2311271316', 'dlr_err': '000', 'dlr_text': '', 'dlr_id': '7306068265450541436', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:18 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': '7306068265456767356', 'headers': {'type': 'deliver_sm', 'cid': 'silknet', 'dlr_dlvrd': '001', 'dlr_sub': '001', 'dlr_sdate': '2311271316', 'dlr_ddate': '2311271316', 'dlr_err': '000', 'dlr_text': '', 'dlr_id': '7306068265456767356', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:18 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': '7306068265450803580', 'headers': {'type': 'deliver_sm', 'cid': 'silknet', 'dlr_dlvrd': '001', 'dlr_sub': '001', 'dlr_sdate': '2311271316', 'dlr_ddate': '2311271316', 'dlr_err': '000', 'dlr_text': '', 'dlr_id': '7306068265450803580', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:18 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000030141213260', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000030141213260', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:19 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': '7306068265456636284', 'headers': {'type': 'deliver_sm', 'cid': 'silknet', 'dlr_dlvrd': '001', 'dlr_sub': '001', 'dlr_sdate': '2311271316', 'dlr_ddate': '2311271316', 'dlr_err': '000', 'dlr_text': '', 'dlr_id': '7306068265456636284', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:19 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000029186752070', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000029186752070', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:19 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000029186752077', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000029186752077', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:19 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': '7306068265450672508', 'headers': {'type': 'deliver_sm', 'cid': 'silknet', 'dlr_dlvrd': '001', 'dlr_sub': '001', 'dlr_sdate': '2311271316', 'dlr_ddate': '2311271316', 'dlr_err': '000', 'dlr_text': '', 'dlr_id': '7306068265450672508', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:19 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000029186752058', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000029186752058', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:19 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000030141213256', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000030141213256', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:19 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': '7306068265453425020', 'headers': {'type': 'deliver_sm', 'cid': 'silknet', 'dlr_dlvrd': '001', 'dlr_sub': '001', 'dlr_sdate': '2311271316', 'dlr_ddate': '2311271316', 'dlr_err': '000', 'dlr_text': '', 'dlr_id': '7306068265453425020', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:19 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000029186752085', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000029186752085', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:19 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000029186752075', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000029186752075', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:19 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': '7306068265455980924', 'headers': {'type': 'deliver_sm', 'cid': 'silknet', 'dlr_dlvrd': '001', 'dlr_sub': '001', 'dlr_sdate': '2311271316', 'dlr_ddate': '2311271316', 'dlr_err': '000', 'dlr_text': '', 'dlr_id': '7306068265455980924', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:19 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000030141213242', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000030141213242', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:19 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000029186752081', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000029186752081', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:19 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000029186752062', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000029186752062', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:19 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': '7306068265456374140', 'headers': {'type': 'deliver_sm', 'cid': 'silknet', 'dlr_dlvrd': '001', 'dlr_sub': '001', 'dlr_sdate': '2311271316', 'dlr_ddate': '2311271316', 'dlr_err': '000', 'dlr_text': '', 'dlr_id': '7306068265456374140', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:19 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000030141213240', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000030141213240', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:19 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000029186752064', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000029186752064', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:19 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000029186752083', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000029186752083', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:19 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='EXPIRED', children=[], properties={'message-id': 'SM000000030135747971', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000030135747971', 'dlr_stat': 'EXPIRED'}}>}
2023-11-27 09:16:19 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000029186752073', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000029186752073', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:20 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000030141213231', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000030141213231', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:20 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': '7306068265456111996', 'headers': {'type': 'deliver_sm', 'cid': 'silknet', 'dlr_dlvrd': '001', 'dlr_sub': '001', 'dlr_sdate': '2311271316', 'dlr_ddate': '2311271316', 'dlr_err': '000', 'dlr_text': '', 'dlr_id': '7306068265456111996', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:20 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000030141213272', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000030141213272', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:20 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000030141213268', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000030141213268', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:20 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000029186752079', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000029186752079', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:20 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000030141213262', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000030141213262', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:24 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000030141213258', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000030141213258', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:27 INFO     1 Connecting to IPv4Address(type='TCP', host='rabbit-mq', port=5672) ...
2023-11-27 09:16:27 INFO     1 Connection made to rabbit-mq:5672
2023-11-27 09:16:27 ERROR    1 AMQP Client is not connected, cannot publish: {'exchange': 'messaging', 'routing_key': 'dlr.deliver_sm', 'content': <DLR instance: body='DELIVRD', children=[], properties={'message-id': 'SM000000030141213253', 'headers': {'type': 'deliver_sm', 'cid': 'magti2', 'dlr_dlvrd': 'ND', 'dlr_sub': 'ND', 'dlr_sdate': 'ND', 'dlr_ddate': 'ND', 'dlr_err': 'ND', 'dlr_text': '', 'dlr_id': 'SM000000030141213253', 'dlr_stat': 'DELIVRD'}}>}
2023-11-27 09:16:27 INFO     1 Successfull authentication
2023-11-27 09:16:27 INFO     1 Got channel
2023-11-27 09:16:27 INFO     1 The channel is open
2023-11-27 09:16:27 ERROR    1 Channel open failed: [Failure instance: Traceback: <class 'twisted.internet.defer.AlreadyCalledError'>: 
/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py:1750:gotResult
/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py:1737:_inlineCallbacks
/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py:661:callback
/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py:763:_startRunCallbacks
--- <exception caught here> ---
/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py:857:_runCallbacks
/usr/local/lib/python3.10/site-packages/jasmin/queues/factory.py:181:_channel_open
/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py:661:callback
/usr/local/lib/python3.10/site-packages/twisted/internet/defer.py:756:_startRunCallbacks
]
2023-11-27 09:36:04 ERROR    1 Connection lost. Reason: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion: Connection lost.
]
2023-11-27 09:36:04 INFO     1 Reconnecting after 10 seconds ...
2023-11-27 09:37:44 INFO     1 Connecting to IPv4Address(type='TCP', host='rabbit-mq', port=5672) ...
2023-11-27 09:37:44 INFO     1 Connection made to rabbit-mq:5672
2023-11-27 09:37:44 INFO     1 Successfull authentication
2023-11-27 09:37:44 INFO     1 Got channel
2023-11-27 09:37:44 INFO     1 The channel is open
2023-11-27 09:37:44 INFO     1 A new queue has been successfully declared [RouterPB_deliver_sm_all]
2023-11-27 09:37:44 INFO     1 A new queue has been successfully declared [RouterPB_bill_request_submit_sm_resp_all]
2023-11-27 09:37:44 INFO     1 A new queue has been successfully declared [DLRLookup-main]
2023-11-27 09:37:44 INFO     1 A new queue has been successfully declared [deliver_sm_thrower]
2023-11-27 09:37:44 INFO     1 A new queue has been successfully declared [dlr_thrower]
2023-11-27 09:37:44 INFO     1 A new queue has been successfully declared [submit.sm.magti]
2023-11-27 09:37:44 INFO     1 A new queue has been successfully declared [submit.sm.silknet]
2023-11-27 09:37:44 INFO     1 A new queue has been successfully declared [submit.sm.cellfie]
2023-11-27 09:37:44 INFO     1 A new queue has been successfully declared [submit.sm.magti2]
2023-11-27 09:37:44 INFO     1 A new queue has been successfully declared [submit.sm.silknetapi]
2023-11-30 18:29:29 ERROR    1 Connection lost. Reason: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion: Connection lost.
]
2023-11-30 18:29:29 INFO     1 Reconnecting after 10 seconds ...
2023-11-30 18:30:33 INFO     1 Connecting to IPv4Address(type='TCP', host='rabbit-mq', port=5672) ...
2023-11-30 18:30:33 INFO     1 Connection made to rabbit-mq:5672
2023-11-30 18:30:33 INFO     1 Successfull authentication
2023-11-30 18:30:33 INFO     1 Got channel
2023-11-30 18:30:33 INFO     1 The channel is open
2023-11-30 18:30:33 INFO     1 A new queue has been successfully declared [RouterPB_deliver_sm_all]
2023-11-30 18:30:33 INFO     1 A new queue has been successfully declared [RouterPB_bill_request_submit_sm_resp_all]
2023-11-30 18:30:33 INFO     1 A new queue has been successfully declared [DLRLookup-main]
2023-11-30 18:30:33 INFO     1 A new queue has been successfully declared [deliver_sm_thrower]
2023-11-30 18:30:33 INFO     1 A new queue has been successfully declared [dlr_thrower]
2023-11-30 18:30:33 INFO     1 A new queue has been successfully declared [submit.sm.magti]
2023-11-30 18:30:33 INFO     1 A new queue has been successfully declared [submit.sm.silknet]
2023-11-30 18:30:33 INFO     1 A new queue has been successfully declared [submit.sm.cellfie]
2023-11-30 18:30:33 INFO     1 A new queue has been successfully declared [submit.sm.magti2]
2023-11-30 18:30:33 INFO     1 A new queue has been successfully declared [submit.sm.silknetapi]
2023-12-02 11:59:57 ERROR    1 Connection lost. Reason: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion: Connection lost.
]
2023-12-02 11:59:57 INFO     1 Reconnecting after 10 seconds ...
2023-12-02 12:00:39 INFO     1 Connecting to IPv4Address(type='TCP', host='rabbit-mq', port=5672) ...
2023-12-02 12:00:39 ERROR    1 Connection failed. Reason: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionRefusedError'>: Connection was refused by other side: 111: Connection refused.
]
2023-12-02 12:00:39 INFO     1 Reconnecting after 10 seconds ...
2023-12-02 12:00:49 INFO     1 Connecting to IPv4Address(type='TCP', host='rabbit-mq', port=5672) ...
2023-12-02 12:00:49 ERROR    1 Connection failed. Reason: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionRefusedError'>: Connection was refused by other side: 111: Connection refused.
]
2023-12-02 12:00:49 INFO     1 Reconnecting after 10 seconds ...
2023-12-02 12:00:59 INFO     1 Connecting to IPv4Address(type='TCP', host='rabbit-mq', port=5672) ...
2023-12-02 12:00:59 INFO     1 Connection made to rabbit-mq:5672
2023-12-02 12:00:59 INFO     1 Successfull authentication
2023-12-02 12:00:59 INFO     1 Got channel
2023-12-02 12:00:59 INFO     1 The channel is open
2023-12-02 12:00:59 INFO     1 A new queue has been successfully declared [RouterPB_deliver_sm_all]
2023-12-02 12:00:59 INFO     1 A new queue has been successfully declared [RouterPB_bill_request_submit_sm_resp_all]
2023-12-02 12:00:59 INFO     1 A new queue has been successfully declared [DLRLookup-main]
2023-12-02 12:00:59 INFO     1 A new queue has been successfully declared [deliver_sm_thrower]
2023-12-02 12:00:59 INFO     1 A new queue has been successfully declared [dlr_thrower]
2023-12-02 12:00:59 INFO     1 A new queue has been successfully declared [submit.sm.magti]
2023-12-02 12:00:59 INFO     1 A new queue has been successfully declared [submit.sm.silknet]
2023-12-02 12:00:59 INFO     1 A new queue has been successfully declared [submit.sm.cellfie]
2023-12-02 12:00:59 INFO     1 A new queue has been successfully declared [submit.sm.magti2]
2023-12-02 12:00:59 INFO     1 A new queue has been successfully declared [submit.sm.silknetapi]
2023-12-02 13:16:40 ERROR    1 Connection lost. Reason: [Failure instance: Traceback (failure with no frames): <class 'twisted.internet.error.ConnectionLost'>: Connection to the other side was lost in a non-clean fashion: Connection lost.
]
2023-12-02 13:16:40 INFO     1 Reconnecting after 10 seconds ...
2023-12-02 13:17:29 INFO     1 Connecting to IPv4Address(type='TCP', host='rabbit-mq', port=5672) ...
2023-12-02 13:17:29 INFO     1 Connection made to rabbit-mq:5672
2023-12-02 13:17:29 INFO     1 Successfull authentication
2023-12-02 13:17:29 INFO     1 Got channel
2023-12-02 13:17:29 INFO     1 The channel is open
2023-12-02 13:17:29 INFO     1 A new queue has been successfully declared [RouterPB_deliver_sm_all]
2023-12-02 13:17:29 INFO     1 A new queue has been successfully declared [RouterPB_bill_request_submit_sm_resp_all]
2023-12-02 13:17:29 INFO     1 A new queue has been successfully declared [DLRLookup-main]
2023-12-02 13:17:29 INFO     1 A new queue has been successfully declared [deliver_sm_thrower]
2023-12-02 13:17:29 INFO     1 A new queue has been successfully declared [dlr_thrower]
2023-12-02 13:17:29 INFO     1 A new queue has been successfully declared [submit.sm.magti]
2023-12-02 13:17:29 INFO     1 A new queue has been successfully declared [submit.sm.silknet]
2023-12-02 13:17:29 INFO     1 A new queue has been successfully declared [submit.sm.cellfie]
2023-12-02 13:17:29 INFO     1 A new queue has been successfully declared [submit.sm.magti2]
2023-12-02 13:17:29 INFO     1 A new queue has been successfully declared [submit.sm.silknetapi]

@BlackOrder
Copy link
Contributor

i think you are setting the heartbeat = 1 is not good. any latency and rabbitmq will consider the connection dead. rabbitmq can see this setting and expect the heartbeat. if there is any delay for whatever reason, your connection is considered dead from rabbit side.
set heartbeat to 30s or more.
it is meant to keep the connection active, not a connection failure detection system.
If you want to detect if jasmin lost the connection to rabbitmq, check this healthCheck settings:


    healthcheck:
      test: ["CMD-SHELL", "wget -qO- --header=\"Authorization: Basic $(echo -n ${AMQP_BROKER_USERNAME:-guest}:${AMQP_BROKER_PASSWORD:-guest} | base64)\" \"http://${AMQP_BROKER_HOST:-rabbit-mq}:${AMQP_BROKER_HTTP_PORT:-15672}/api/queues/%2F/RouterPB_bill_request_submit_sm_resp_all\" | grep -o '\"consumer_tag\":\\s*\"RouterPB-billrequests\"' | wc -l | grep -q 0 && (bash -c 'kill -s 15 1') && exit 1 || exit 0"]
      interval: 15s
      timeout: 10s
      retries: 1
      start_period: 20s

i do a check on the RouterPB-billrequests queue if it still have a consumer. you can test for lower intervals, but even in my local docker the connection takes 8-14 seconds to complete.

@michaeltoop
Copy link

Hi all. We are having the same issue (Jasmin does not try to reconnect after a disconnection from Rabbit). We observed this yesterday when Rabbit restarted for maintenance).

@Osa1989
Copy link

Osa1989 commented Mar 11, 2024

Hi all, I am also having the same issue, any idea how to solve it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@farirat @BlackOrder @marcinkowalczyk @Osa1989 @michaeltoop @vaska94 @aivisol