recv() call blocked indefinitely in Net::AMQP::RabbitMQ

[kaniMano]

Describe the bug

We could see the one of the process using rabbitmq got struck in recv() call and not coming out of it.

/var/log # pstack 2629417
#0 0x00007fb330f02cee in recv () from /lib64/libpthread.so.0
#1 0x00007fb325d897ed in amqp_tcp_socket_recv () from /opt/airwave/perlbrew/cpan/5.10/lib/perl5/x86_64-linux-thread-multi/auto/Net/AMQP/RabbitMQ/RabbitMQ.so
#2 0x00007fb325d87028 in recv_with_timeout () from /opt/airwave/perlbrew/cpan/5.10/lib/perl5/x86_64-linux-thread-multi/auto/Net/AMQP/RabbitMQ/RabbitMQ.so
#3 0x00007fb325d87973 in wait_frame_inner () from /opt/airwave/perlbrew/cpan/5.10/lib/perl5/x86_64-linux-thread-multi/auto/Net/AMQP/RabbitMQ/RabbitMQ.so
#4 0x00007fb325d7c6d2 in internal_recv () from /opt/airwave/perlbrew/cpan/5.10/lib/perl5/x86_64-linux-thread-multi/auto/Net/AMQP/RabbitMQ/RabbitMQ.so
#5 0x00007fb325d7dafd in XS_Net__AMQP__RabbitMQ_recv () from /opt/airwave/perlbrew/cpan/5.10/lib/perl5/x86_64-linux-thread-multi/auto/Net/AMQP/RabbitMQ/RabbitMQ.so
#6 0x00007fb32f8c4d9f in Perl_pp_entersub () from /etc/httpd/modules/mod_perl.so
#7 0x00007fb32f8bcb30 in Perl_runops_standard () from /etc/httpd/modules/mod_perl.so
#8 0x00007fb32f866044 in Perl_call_sv () from /etc/httpd/modules/mod_perl.so
#9 0x00007fb31bcfb81c in e_cb () from /opt/airwave/perlbrew/cpan/5.10/lib/perl5/x86_64-linux-thread-multi/auto/EV/EV.so
#10 0x00007fb31bcf6857 in ev_invoke_pending () from /opt/airwave/perlbrew/cpan/5.10/lib/perl5/x86_64-linux-thread-multi/auto/EV/EV.so
#11 0x00007fb31bd137ad in ev_run () from /opt/airwave/perlbrew/cpan/5.10/lib/perl5/x86_64-linux-thread-multi/auto/EV/EV.so
#12 0x00007fb31bd14a8f in XS_EV_run () from /opt/airwave/perlbrew/cpan/5.10/lib/perl5/x86_64-linux-thread-multi/auto/EV/EV.so
#13 0x00007fb32f8c4d9f in Perl_pp_entersub () from /etc/httpd/modules/mod_perl.so
#14 0x00007fb32f8bcb30 in Perl_runops_standard () from /etc/httpd/modules/mod_perl.so
#15 0x00007fb32f8666bc in Perl_eval_sv () from /etc/httpd/modules/mod_perl.so
#16 0x00007fb32f8685f8 in Perl_require_pv () from /etc/httpd/modules/mod_perl.so
#17 0x00007fb32f840e1e in modperl_require_file () from /etc/httpd/modules/mod_perl.so
#18 0x00007fb32f83b4de in modperl_config_apply_PerlRequire () from /etc/httpd/modules/mod_perl.so
#19 0x00007fb32f839068 in modperl_startup () from /etc/httpd/modules/mod_perl.so
#20 0x00007fb32f838e50 in modperl_startup () from /etc/httpd/modules/mod_perl.so
#21 0x00007fb32f8392d6 in modperl_init () from /etc/httpd/modules/mod_perl.so
#22 0x00007fb32f839436 in modperl_hook_init () from /etc/httpd/modules/mod_perl.so
#23 0x00005650385cc6d3 in ap_run_open_logs ()
#24 0x00005650385aac01 in main ()

Reproduction steps

We are using RHEL platform, we are seeing this issue randomly. Only killing the service is helping out.

Expected behavior

#0 0x00007fb330a5715b in epoll_wait () from /lib64/libc.so.6
#1 0x00007fb31bd0dec5 in epoll_poll () from /opt/airwave/perlbrew/cpan/5.10/lib/perl5/x86_64-linux-thread-multi/auto/EV/EV.so
#2 0x00007fb31bd132e8 in ev_run () from /opt/airwave/perlbrew/cpan/5.10/lib/perl5/x86_64-linux-thread-multi/auto/EV/EV.so
#3 0x00007fb31bd14a8f in XS_EV_run () from /opt/airwave/perlbrew/cpan/5.10/lib/perl5/x86_64-linux-thread-multi/auto/EV/EV.so
#4 0x00007fb32f8c4d9f in Perl_pp_entersub () from /etc/httpd/modules/mod_perl.so
#5 0x00007fb32f8bcb30 in Perl_runops_standard () from /etc/httpd/modules/mod_perl.so
#6 0x00007fb32f8666bc in Perl_eval_sv () from /etc/httpd/modules/mod_perl.so
#7 0x00007fb32f8685f8 in Perl_require_pv () from /etc/httpd/modules/mod_perl.so
#8 0x00007fb32f840e1e in modperl_require_file () from /etc/httpd/modules/mod_perl.so
#9 0x00007fb32f83b4de in modperl_config_apply_PerlRequire () from /etc/httpd/modules/mod_perl.so
#10 0x00007fb32f839068 in modperl_startup () from /etc/httpd/modules/mod_perl.so
#11 0x00007fb32f838e50 in modperl_startup () from /etc/httpd/modules/mod_perl.so
#12 0x00007fb32f8392d6 in modperl_init () from /etc/httpd/modules/mod_perl.so
#13 0x00007fb32f839436 in modperl_hook_init () from /etc/httpd/modules/mod_perl.so
#14 0x00005650385cc6d3 in ap_run_open_logs ()
#15 0x00005650385aac01 in main ()
this should be the correct trace in working condition

Additional context

No response

[kaniMano]

I tried to pass timeout to recv() for 1000ms , but it always fails if any argument is given.
error Usage: Net::AMQP::RabbitMQ::recv(conn)

[michaelklishin]

@kaniMano our team does not maintain that Perl client. In fact, I'm not even sure it is one of the clients we list on our Developer Tools page.

I'm not saying that there's anything inherently wrong with this client or that you should switch but this is not the right place to get help with Perl clients.

[lukebakken]

@kaniMano this would be the place to report this issue - https://github.com/net-amqp-rabbitmq/net-amqp-rabbitmq

However, the information you provide is woefully insufficient for someone to assist you. You would have to provide code to reproduce it to have a chance of getting assistance.