rserver can crash when proxying traffic to a websocket

[jmcphers]

System details

OS Version      : RHEL 7.3
RStudio Version : 1.1.383
R Version       : Any

Steps to reproduce the problem

1. Use Shiny applications in RStudio Server
2. [unknown]

Describe the problem in detail

Many customers are seeing RStudio Server crashes with the following log entry:

Jan 15 10:34:04  kernel: rserver[26589]: segfault at 90 ip 000000000064508d sp 00007f50d5ff6660 error 4 in rserver[400000+73e000]

We were able to collect the following crash log from one such customer:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f5b2f7fe700 (LWP 26932)]
0x000000000064508d in rstudio_boost::asio::detail::epoll_reactor::deregister_descriptor(int, rstudio_boost::asio::detail::epoll_reactor::descriptor_state*&, bool) ()
(gdb) i sh
From                To                  Syms Read   Shared Object Library
0x00007f5b39091e60  0x00007f5b3909295e  Yes (*)     /lib64/libdl.so.2
0x00007f5b38e7a900  0x00007f5b38e85ce1  Yes (*)     /lib64/libpthread.so.0
0x00007f5b38c72ea0  0x00007f5b38c73794  Yes (*)     /lib64/libutil.so.1
0x00007f5b38a6e510  0x00007f5b38a6fb62  Yes (*)     /lib64/libuuid.so.1
0x00007f5b38867250  0x00007f5b3886a04c  Yes (*)     /lib64/librt.so.1
0x00007f5b38651170  0x00007f5b3865d6f8  Yes (*)     /lib64/libz.so.1
0x00007f5b383f7bb0  0x00007f5b3843358d  Yes (*)     /lib64/libssl.so.10
0x00007f5b37fe8f00  0x00007f5b38120bd7  Yes (*)     /lib64/libcrypto.so.10
0x00007f5b37d6f670  0x00007f5b37d76d0c  Yes (*)     /lib64/libpam.so.0
0x00007f5b37ac0510  0x00007f5b37b275aa  Yes (*)     /lib64/libstdc++.so.6
0x00007f5b37768370  0x00007f5b377d3276  Yes (*)     /lib64/libm.so.6
0x00007f5b3754faf0  0x00007f5b3755f295  Yes (*)     /lib64/libgcc_s.so.1
0x00007f5b371a9480  0x00007f5b372efbcf  Yes (*)     /lib64/libc.so.6
0x00007f5b39295b10  0x00007f5b392b0710  Yes (*)     /lib64/ld-linux-x86-64.so.2
0x00007f5b36f49650  0x00007f5b36f7ba1a  Yes (*)     /lib64/libgssapi_krb5.so.2
0x00007f5b36c7aa10  0x00007f5b36ce1e8a  Yes (*)     /lib64/libkrb5.so.3
0x00007f5b36a52570  0x00007f5b36a53144  Yes (*)     /lib64/libcom_err.so.2
0x00007f5b368228c0  0x00007f5b36840c0f  Yes (*)     /lib64/libk5crypto.so.3
0x00007f5b365f9040  0x00007f5b365fee52  Yes (*)     /lib64/libaudit.so.1
0x00007f5b363eb890  0x00007f5b363f242b  Yes (*)     /lib64/libkrb5support.so.0
0x00007f5b361e55b0  0x00007f5b361e61cc  Yes (*)     /lib64/libkeyutils.so.1
0x00007f5b35fcd9d0  0x00007f5b35fdc7e1  Yes (*)     /lib64/libresolv.so.2
0x00007f5b35dc5580  0x00007f5b35dc73bc  Yes (*)     /lib64/libcap-ng.so.0
0x00007f5b35ba3ac0  0x00007f5b35bb98a6  Yes (*)     /lib64/libselinux.so.1
0x00007f5b3593d5f0  0x00007f5b359828b0  Yes (*)     /lib64/libpcre.so.1
0x00007f5b3572b1d0  0x00007f5b357323e1  Yes (*)     /lib64/libnss_files.so.2
0x00007f5b355215f0  0x00007f5b35526245  Yes (*)     /lib64/libnss_sss.so.2
(*): Shared library is missing debugging information.
(gdb) bt
#0  0x000000000064508d in rstudio_boost::asio::detail::epoll_reactor::deregister_descriptor(int, rstudio_boost::asio::detail::epoll_reactor::descriptor_state*&, bool) ()
#1  0x00000000006495e9 in rstudio::core::Error rstudio::core::http::closeSocket<rstudio_boost::asio::basic_stream_socket<rstudio_boost::asio::local::stream_protocol, rstudio_boost::asio::stream_socket_service<rstudio_boost::asio::local::stream_protocol> > >(rstudio_boost::asio::basic_stream_socket<rstudio_boost::asio::local::stream_protocol, rstudio_boost::asio::stream_socket_service<rstudio_boost::asio::local::stream_protocol> >&) ()
#2  0x000000000064a284 in rstudio::core::http::AsyncConnectionImpl<rstudio_boost::asio::local::stream_protocol>::close() ()
#3  0x000000000086d894 in rstudio::core::http::SocketProxy::handleError(rstudio_boost::system::error_code const&, rstudio::core::ErrorLocation const&) ()
#4  0x000000000086dba8 in rstudio::core::http::SocketProxy::handleClientWrite(rstudio_boost::system::error_code const&, unsigned long) ()
#5  0x0000000000645a58 in rstudio_boost::function2<void, rstudio_boost::system::error_code const&, unsigned long>::operator()(rstudio_boost::system::error_code const&, unsigned long) const ()
#6  0x0000000000653853 in rstudio_boost::asio::detail::write_op<rstudio_boost::asio::basic_stream_socket<rstudio_boost::asio::local::stream_protocol, rstudio_boost::asio::stream_socket_service<rstudio_boost::asio::local::stream_protocol> >, std::vector<rstudio_boost::asio::const_buffer, std::allocator<rstudio_boost::asio::const_buffer> >, rstudio_boost::asio::detail::transfer_all_t, rstudio_boost::function<void (rstudio_boost::system::error_code const&, unsigned long)> >::operator()(rstudio_boost::system::error_code const&, unsigned long, int) ()
#7  0x0000000000654559 in rstudio_boost::asio::detail::reactive_socket_send_op<rstudio_boost::asio::detail::consuming_buffers<rstudio_boost::asio::const_buffer, std::vector<rstudio_boost::asio::const_buffer, std::allocator<rstudio_boost::asio::const_buffer> > >, rstudio_boost::asio::detail::write_op<rstudio_boost::asio::basic_stream_socket<rstudio_boost::asio::local::stream_protocol, rstudio_boost::asio::stream_socket_service<rstudio_boost::asio::local::stream_protocol> >, std::vector<rstudio_boost::asio::const_buffer, std::allocator<rstudio_boost::asio::const_buffer> >, rstudio_boost::asio::detail::transfer_all_t, rstudio_boost::function<void (rstudio_boost::system::error_code const&, unsigned long)> > >::do_complete(rstudio_boost::asio::detail::task_io_service*, rstudio_boost::asio::detail::task_io_service_operation*, rstudio_boost::system::error_code const&, unsigned long) ()
#8  0x000000000064b0b8 in rstudio_boost::asio::detail::task_io_service::run(rstudio_boost::system::error_code&) ()
#9  0x000000000064b486 in rstudio::core::http::AsyncServerImpl<rstudio_boost::asio::local::stream_protocol>::runServiceThread() ()
#10 0x000000000095ec6b in thread_proxy ()
#11 0x00007f5b38e7ce25 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f5b3728234d in clone () from /lib64/libc.so.6

This trace suggests that an error while closing the websocket that rserver is proxying to is crashing the rserver process; the customer supplying has noted that these crashes seem to correlate with running Shiny applications inside RStudio Server, but it doesn't happen every time (or even most the time).

However, it bears investigating as we see a lot of rserver segfaults in customer logs and while it's possible that they're all caused by different things it's more likely that there are one or two issues responsible for the majority of the crashes, of which this may possible be one.

See internal ticket 24819 for an example.

[kevinushey]

Potentially related: https://svn.boost.org/trac10/ticket/7611.

[kfeinauer]

Repro steps:

1. Run a shiny app in external mode (new browser tab) via the Run App button
2. Run several additional apps by continuing to click the Run App button
3. Close all open shiny app tabs that were opened in step 2
4. Continuously open a new app tab by click Run App and close it, and repeat until the rserver process crashes.

[jmcphers]

Thanks @kfeinauer, having a reliable repro for this is invaluable! We see rserver segfaults all the time in customer installations and several have been explicitly correlated with Shiny (perhaps others are implicitly). Let's try to fix this in 1.2.