Help debugging strange node crashes with quorum queues #10679

tkoft · 2024-03-05T16:28:51Z

tkoft
Mar 5, 2024

I've got a cluster deployed with three nodes that went into a weird state this morning when there shouldn't have been any traffic on the cluster at all.

Having a hard time figuring out what's going on at all, but it seems to be related to quorum queues? Using RabbitMQ 3.11.18 management image deployed with kubernetes cluster operator.

Here's a dump of the logs from the first crash, but it appears the node kept retrying and crashing (with successful client connections getting logged in between). The quorum queue ended up in a bad state (minority in cluster) even though two other nodes were supposedly up.

2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118> ** Generic server <0.19440.118> terminating
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118> ** Last message in was {'$gen_cast',
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                            {method,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                                {'basic.consume',0,<<"accounts_events">>,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                                    <<"None1">>,false,false,false,false,[]},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                                none,noflow}}
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118> ** When Server state == {ch,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                          {conf,running,rabbit_framing_amqp_0_9_1,1,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                           <0.19433.118>,<0.19436.118>,<0.19433.118>,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                           <<"10.227.1.39:54170 -> 10.227.0.199:5672">>,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                           undefined,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                           {user,<<"default_user_RXWP6jnJNs8x0hDGacN">>,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                            [administrator],
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                            [{rabbit_auth_backend_internal,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                              #Fun<rabbit_auth_backend_internal.3.131409679>}]},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                           <<"venuesync">>,<<"accounts_events">>,<0.19434.118>,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                           [{<<"consumer_cancel_notify">>,bool,true},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                            {<<"connection.blocked">>,bool,true},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                            {<<"authentication_failure_close">>,bool,true}],
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                           none,0,134217728,1800000,#{},1000000000},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                          {lstate,<0.19435.118>,false},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                          none,1,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                          {0,[],[]},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                          {state,#{},erlang},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                          #{},#{},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                          {state,fine,5000,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                           #Ref<0.1905515858.2512912385.164784>},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                          false,1,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                          {rabbit_confirms,undefined,#{}},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                          [],[],none,flow,[],
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                          {rabbit_queue_type,#{},#{}},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                          #Ref<0.1905515858.2512912385.164780>,false}
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118> ** Reason for termination ==
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118> ** {{badmatch,{error,noproc}},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>     [{rabbit_quorum_queue,consume,3,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                           [{file,"rabbit_quorum_queue.erl"},{line,791}]},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>      {rabbit_queue_type,consume,3,[{file,"rabbit_queue_type.erl"},{line,408}]},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>      {rabbit_channel,'-basic_consume/8-fun-0-',10,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                      [{file,"rabbit_channel.erl"},{line,1803}]},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>      {rabbit_misc,with_exit_handler,2,[{file,"rabbit_misc.erl"},{line,531}]},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>      {rabbit_channel,basic_consume,8,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                      [{file,"rabbit_channel.erl"},{line,1800}]},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>      {rabbit_channel,handle_method,3,
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>                      [{file,"rabbit_channel.erl"},{line,1479}]},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>      {rabbit_channel,handle_cast,2,[{file,"rabbit_channel.erl"},{line,647}]},
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118>      {gen_server2,handle_msg,2,[{file,"gen_server2.erl"},{line,1067}]}]}
2024-03-05 13:42:28.692815+00:00 [error] <0.19440.118> 
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>   crasher:
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>     initial call: rabbit_channel:init/1
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>     pid: <0.19440.118>
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>     registered_name: []
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>     exception exit: {{badmatch,{error,noproc}},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                      [{rabbit_quorum_queue,consume,3,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                           [{file,"rabbit_quorum_queue.erl"},{line,791}]},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                       {rabbit_queue_type,consume,3,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                           [{file,"rabbit_queue_type.erl"},{line,408}]},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                       {rabbit_channel,'-basic_consume/8-fun-0-',10,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                           [{file,"rabbit_channel.erl"},{line,1803}]},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                       {rabbit_misc,with_exit_handler,2,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                           [{file,"rabbit_misc.erl"},{line,531}]},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                       {rabbit_channel,basic_consume,8,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                           [{file,"rabbit_channel.erl"},{line,1800}]},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                       {rabbit_channel,handle_method,3,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                           [{file,"rabbit_channel.erl"},{line,1479}]},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                       {rabbit_channel,handle_cast,2,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                           [{file,"rabbit_channel.erl"},{line,647}]},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                       {gen_server2,handle_msg,2,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                           [{file,"gen_server2.erl"},{line,1067}]}]}
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>       in function  gen_server2:terminate/3 (gen_server2.erl, line 1183)
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>     ancestors: [<0.19439.118>,<0.19437.118>,<0.19432.118>,<0.19431.118>,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                   <0.841.0>,<0.840.0>,<0.839.0>,<0.837.0>,<0.836.0>,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                   rabbit_sup,<0.230.0>]
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>     message_queue_len: 4
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>     messages: [{'$gen_cast',{command,{'basic.consume_ok',<<"None1">>}}},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                   emit_stats,emit_gen_server2_stats,tick]
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>     links: [<0.19439.118>]
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>     dictionary: [{permission_cache,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                       [{{resource,<<"venuesync">>,queue,<<"accounts_events">>},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                         #{},read},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                        {{resource,<<"venuesync">>,exchange,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                             <<"venuesync_events_consumer">>},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                         #{},read},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                        {{resource,<<"venuesync">>,queue,<<"accounts_events">>},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                         #{},write},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                        {{resource,<<"venuesync">>,queue,<<"accounts_events">>},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                         #{},configure},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                        {{resource,<<"venuesync">>,exchange,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                             <<"venuesync_events_consumer">>},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                         #{},configure}]},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                   {rand_seed,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                       {#{jump => #Fun<rand.3.34006561>,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                          max => 288230376151711743,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                          next => #Fun<rand.5.34006561>,type => exsplus},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                        [117114323147441561|99075661864211887]}},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                   {process_name,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                       {rabbit_channel,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                           {<<"10.227.1.39:54170 -> 10.227.0.199:5672">>,1}}},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                   {permission_cache_can_expire,false},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                   {{xtype_to_module,topic},rabbit_exchange_type_topic},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                   {topic_permission_cache,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                       [{{resource,<<"venuesync">>,topic,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                             <<"venuesync_events_consumer">>},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                         #{routing_key => <<"*.users.*">>,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                           variable_map =>
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                               #{<<"username">> =>
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                                     <<"default_user_RXWP6jnJNs8x0hDGacN">>,
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                                 <<"vhost">> => <<"venuesync">>}},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                         read}]},
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>                   {channel_operation_timeout,15000}]
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>     trap_exit: true
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>     status: running
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>     heap_size: 17731
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>     stack_size: 28
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>     reductions: 37673
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118>   neighbours:
2024-03-05 13:42:28.694041+00:00 [error] <0.19440.118> 
2024-03-05 13:42:28.696155+00:00 [error] <0.19433.118> Error on AMQP connection <0.19433.118> (10.227.1.39:54170 -> 10.227.0.199:5672, vhost: 'venuesync', user: 'default_user_RXWP6jnJNs8x0hDGacN', state: running), channel 1:
2024-03-05 13:42:28.696155+00:00 [error] <0.19433.118>  {{badmatch,{error,noproc}},
2024-03-05 13:42:28.696155+00:00 [error] <0.19433.118>  [{rabbit_quorum_queue,consume,3,
2024-03-05 13:42:28.696155+00:00 [error] <0.19433.118>                        [{file,"rabbit_quorum_queue.erl"},{line,791}]},
2024-03-05 13:42:28.696155+00:00 [error] <0.19433.118>   {rabbit_queue_type,consume,3,[{file,"rabbit_queue_type.erl"},{line,408}]},
2024-03-05 13:42:28.696155+00:00 [error] <0.19433.118>   {rabbit_channel,'-basic_consume/8-fun-0-',10,
2024-03-05 13:42:28.696155+00:00 [error] <0.19433.118>                   [{file,"rabbit_channel.erl"},{line,1803}]},
2024-03-05 13:42:28.696155+00:00 [error] <0.19433.118>   {rabbit_misc,with_exit_handler,2,[{file,"rabbit_misc.erl"},{line,531}]},
2024-03-05 13:42:28.696155+00:00 [error] <0.19433.118>   {rabbit_channel,basic_consume,8,[{file,"rabbit_channel.erl"},{line,1800}]},
2024-03-05 13:42:28.696155+00:00 [error] <0.19433.118>   {rabbit_channel,handle_method,3,[{file,"rabbit_channel.erl"},{line,1479}]},
2024-03-05 13:42:28.696155+00:00 [error] <0.19433.118>   {rabbit_channel,handle_cast,2,[{file,"rabbit_channel.erl"},{line,647}]},
2024-03-05 13:42:28.696155+00:00 [error] <0.19433.118>   {gen_server2,handle_msg,2,[{file,"gen_server2.erl"},{line,1067}]}]}
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>     supervisor: {<0.19439.118>,rabbit_channel_sup}
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>     errorContext: child_terminated
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>     reason: {{badmatch,{error,noproc}},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>              [{rabbit_quorum_queue,consume,3,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                                    [{file,"rabbit_quorum_queue.erl"},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                                     {line,791}]},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>               {rabbit_queue_type,consume,3,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                                  [{file,"rabbit_queue_type.erl"},{line,408}]},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>               {rabbit_channel,'-basic_consume/8-fun-0-',10,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                               [{file,"rabbit_channel.erl"},{line,1803}]},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>               {rabbit_misc,with_exit_handler,2,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                            [{file,"rabbit_misc.erl"},{line,531}]},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>               {rabbit_channel,basic_consume,8,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                               [{file,"rabbit_channel.erl"},{line,1800}]},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>               {rabbit_channel,handle_method,3,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                               [{file,"rabbit_channel.erl"},{line,1479}]},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>               {rabbit_channel,handle_cast,2,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                               [{file,"rabbit_channel.erl"},{line,647}]},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>               {gen_server2,handle_msg,2,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                            [{file,"gen_server2.erl"},{line,1067}]}]}
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>     offender: [{pid,<0.19440.118>},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                {id,channel},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                {mfargs,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                    {rabbit_channel,start_link,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                        [1,<0.19433.118>,<0.19436.118>,<0.19433.118>,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                         <<"10.227.1.39:54170 -> 10.227.0.199:5672">>,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                         rabbit_framing_amqp_0_9_1,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                         {user,<<"default_user_RXWP6jnJNs8x0hDGacN">>,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                             [administrator],
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                             [{rabbit_auth_backend_internal,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                                  #Fun<rabbit_auth_backend_internal.3.131409679>}]},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                         <<"venuesync">>,
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                         [{<<"consumer_cancel_notify">>,bool,true},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                          {<<"connection.blocked">>,bool,true},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                          {<<"authentication_failure_close">>,bool,true}],
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                         <0.19434.118>,<0.19435.118>]}},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                {restart_type,intrinsic},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                {shutdown,70000},
2024-03-05 13:42:28.695675+00:00 [error] <0.19439.118>                {child_type,worker}]
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118> Non-AMQP exit reason '{{badmatch,{error,noproc}},
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                        [{rabbit_quorum_queue,consume,3,
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                             [{file,"rabbit_quorum_queue.erl"},{line,791}]},
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                         {rabbit_queue_type,consume,3,
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                             [{file,"rabbit_queue_type.erl"},{line,408}]},
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                         {rabbit_channel,'-basic_consume/8-fun-0-',10,
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                             [{file,"rabbit_channel.erl"},{line,1803}]},
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                         {rabbit_misc,with_exit_handler,2,
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                             [{file,"rabbit_misc.erl"},{line,531}]},
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                         {rabbit_channel,basic_consume,8,
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                             [{file,"rabbit_channel.erl"},{line,1800}]},
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                         {rabbit_channel,handle_method,3,
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                             [{file,"rabbit_channel.erl"},{line,1479}]},
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                         {rabbit_channel,handle_cast,2,
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                             [{file,"rabbit_channel.erl"},{line,647}]},
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                         {gen_server2,handle_msg,2,
2024-03-05 13:42:28.696743+00:00 [warning] <0.19433.118>                             [{file,"gen_server2.erl"},{line,1067}]}]}'
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>     supervisor: {<0.19439.118>,rabbit_channel_sup}
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>     errorContext: shutdown
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>     reason: reached_max_restart_intensity
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>     offender: [{pid,<0.19440.118>},
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                {id,channel},
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                {mfargs,
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                    {rabbit_channel,start_link,
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                        [1,<0.19433.118>,<0.19436.118>,<0.19433.118>,
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                         <<"10.227.1.39:54170 -> 10.227.0.199:5672">>,
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                         rabbit_framing_amqp_0_9_1,
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                         {user,<<"default_user_RXWP6jnJNs8x0hDGacN">>,
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                             [administrator],
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                             [{rabbit_auth_backend_internal,
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                                  #Fun<rabbit_auth_backend_internal.3.131409679>}]},
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                         <<"venuesync">>,
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                         [{<<"consumer_cancel_notify">>,bool,true},
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                          {<<"connection.blocked">>,bool,true},
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                          {<<"authentication_failure_close">>,bool,true}],
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                         <0.19434.118>,<0.19435.118>]}},
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                {restart_type,intrinsic},
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                {shutdown,70000},
2024-03-05 13:42:28.697378+00:00 [error] <0.19439.118>                {child_type,worker}]

Any help with this would be appreciated!

Answered by michaelklishin

Mar 5, 2024

The line in question. It is one of the steps during consumer registration.

I do not see any obvious connection with #8564. There is no queue filtering going on when a consumer is added, the name of the queue is already known (client-provided).

In any case, the response is still: 3.11 is out of community support, and you are ten patches behind in that series alone.

View full answer

tkoft · 2024-03-05T16:48:15Z

tkoft
Mar 5, 2024
Author

We do have classic queues on that node as well. Is this possibly related to this issue fixed in 3.11.19?

Nodes in clusters that had quorum queues and non-mirrored classic queues on stopped (or failed) nodes
could run into an exception.
The same exception could affect rabbitmqctl list_queues.
GitHub issue: #8596

1 reply

michaelklishin Mar 5, 2024
Maintainer

The line in question. It is one of the steps during consumer registration.

I do not see any obvious connection with #8564. There is no queue filtering going on when a consumer is added, the name of the queue is already known (client-provided).

In any case, the response is still: 3.11 is out of community support, and you are ten patches behind in that series alone.

Answer selected by michaelklishin

michaelklishin · 2024-03-05T17:07:58Z

michaelklishin
Mar 5, 2024
Maintainer

RabbitMQ 3.11 is out of community support, and 3.11.18 specifically is ten patch releases behind.

1 reply

tkoft Mar 5, 2024
Author

Got it, will upgrade and report if it happens again. Thanks.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Help debugging strange node crashes with quorum queues #10679

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 2 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

Help debugging strange node crashes with quorum queues #10679

tkoft Mar 5, 2024

Replies: 2 comments · 2 replies

tkoft Mar 5, 2024 Author

michaelklishin Mar 5, 2024 Maintainer

michaelklishin Mar 5, 2024 Maintainer

tkoft Mar 5, 2024 Author

tkoft
Mar 5, 2024

Replies: 2 comments 2 replies

tkoft
Mar 5, 2024
Author

michaelklishin Mar 5, 2024
Maintainer

michaelklishin
Mar 5, 2024
Maintainer

tkoft Mar 5, 2024
Author