uWS::Loop::defer causes crashes (SIGSEGV)

[damirbarr]

I'm using this library in a multithreaded environment.
On open, I'm storing the Loop object:

...
    m_app = std::make_unique<uWS::App>();
    m_app->ws<UserData>(WEB_SOCKET_PATH.data(),
                        uWS::TemplatedApp<false>::WebSocketBehavior<UserData> {.open =
                                                                                       [this](WebSocketType* ws) {
                                                                                           uWS::Loop* loop = uWS::Loop::get();
                                                                                           m_server_loop = loop;
                                                                                           OTTO_LOG(INFO, "WebSocket connection opened");
                                                                                           on_open(ws);
                                                                                       },
...

Then, when I want to send data, I'm using defer:

...
        m_server_loop->defer([this, data]() {
            if (m_socket)
            {
                m_socket->send(data);
            }
...

I'm getting the following crash:

Thread 23 "ato-server" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffd6ffd700 (LWP 150785)]
__GI___pthread_mutex_lock (mutex=0x43525854464195f9) at ../nptl/pthread_mutex_lock.c:67
67	../nptl/pthread_mutex_lock.c: No such file or directory.
(gdb) bt
#0  __GI___pthread_mutex_lock (mutex=0x43525854464195f9) at ../nptl/pthread_mutex_lock.c:67
#1  0x0000555555789ccc in __gthread_mutex_lock (__mutex=0x43525854464195f9) at /usr/include/x86_64-linux-gnu/c++/10/bits/gthr-default.h:749
#2  0x0000555555789e30 in std::mutex::lock (this=0x43525854464195f9) at /usr/include/c++/10/bits/std_mutex.h:100
#3  0x00005555558edc9d in uWS::Loop::defer(ofats::any_invocable<void ()>&&) (this=0x4352585446416579, cb=...) at /home/gabriel/.conan2/p/uwebs4c1955275aef0/p/include/uWebSockets/Loop.h:173
#4  0x00005555558e7230 in WebSocketServerBase::Send (this=0x555556d54260, data="B 6f96ff92b45547b6935535c9d5eb6428")
    at /home/gabriel/workspace/ato-server/src/connection/sockets/websocket_server_base.cpp:60
#5  0x00005555557af956 in StationIdPublisher::send_station_id (this=0x555556d54320) at /home/gabriel/workspace/ato-server/src/external/station_id_publisher.cpp:33
#6  0x00005555557b176e in std::__invoke_impl<void, void (StationIdPublisher::*&)(), StationIdPublisher*&> (
    __f=@0x555556d64720: (void (StationIdPublisher::*)(StationIdPublisher * const)) 0x5555557af7c8 <StationIdPublisher::send_station_id()>, __t=@0x555556d64730: 0x555556d54320)
    at /usr/include/c++/10/bits/invoke.h:73
#7  0x00005555557b16bb in std::__invoke<void (StationIdPublisher::*&)(), StationIdPublisher*&> (
    __fn=@0x555556d64720: (void (StationIdPublisher::*)(StationIdPublisher * const)) 0x5555557af7c8 <StationIdPublisher::send_station_id()>) at /usr/include/c++/10/bits/invoke.h:95
#8  0x00005555557b1625 in std::_Bind<void (StationIdPublisher::*(StationIdPublisher*))()>::__call<void, , 0ul>(std::tuple<>&&, std::_Index_tuple<0ul>) (this=0x555556d64720, __args=...)
    at /usr/include/c++/10/functional:416
#9  0x00005555557b1565 in std::_Bind<void (StationIdPublisher::*(StationIdPublisher*))()>::operator()<, void>() (this=0x555556d64720) at /usr/include/c++/10/functional:499
#10 0x00005555557b1444 in std::__invoke_impl<void, std::_Bind<void (StationIdPublisher::*(StationIdPublisher*))()>&>(std::__invoke_other, std::_Bind<void (StationIdPublisher::*(StationIdPublisher*))()>&) (__f=...) at /usr/include/c++/10/bits/invoke.h:60
#11 0x00005555557b12d5 in std::__invoke_r<void, std::_Bind<void (StationIdPublisher::*(StationIdPublisher*))()>&>(std::_Bind<void (StationIdPublisher::*(StationIdPublisher*))()>&) (__fn=...)
    at /usr/include/c++/10/bits/invoke.h:110
#12 0x00005555557b10c7 in std::_Function_handler<void (), std::_Bind<void (StationIdPublisher::*(StationIdPublisher*))()> >::_M_invoke(std::_Any_data const&) (__functor=...)
    at /usr/include/c++/10/bits/std_function.h:291
#13 0x000055555589d91e in std::function<void ()>::operator()() const (this=0x555556d4b9b0) at /usr/include/c++/10/bits/std_function.h:622
#14 0x00007ffff6faec77 in Common::Time::Timer::handle_timer_expiration (this=0x555556d4b990) at /home/jenkins/workspace/common_conan_develop/src/linux/common/time/timer.cpp:57
#15 0x00007ffff6fae7de in operator() (__closure=0x7fffd6ffca60) at /home/jenkins/workspace/common_conan_develop/src/linux/common/time/timer.cpp:40
#16 0x00007ffff6fb0e77 in std::__invoke_impl<bool, Common::Time::Timer::timer_poll()::<lambda()>&>(std::__invoke_other, struct {...} &) (__f=...) at /usr/include/c++/10/bits/invoke.h:60
#17 0x00007ffff6fb0a04 in std::__invoke_r<bool, Common::Time::Timer::timer_poll()::<lambda()>&>(struct {...} &) (__fn=...) at /usr/include/c++/10/bits/invoke.h:113
#18 0x00007ffff6fb0254 in std::_Function_handler<bool(), Common::Time::Timer::timer_poll()::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/10/bits/std_function.h:291
#19 0x00005555559a4756 in std::function<bool ()>::operator()() const (this=0x7fffd6ffca60) at /usr/include/c++/10/bits/std_function.h:622
#20 0x00007ffff6f7c2a2 in Common::Polling::Poll(std::function<bool ()> const&, std::function<int ()> const&, std::chrono::duration<long, std::ratio<1l, 1000l> > const&, std::function<bool ()> const&, std::function<bool ()> const&, short) (kill_functor=..., fd_getter=..., listen_interval=std::chrono::duration = { 250ms }, default_callback=..., timeout_callback=..., fd_event=1)
    at /home/jenkins/workspace/common_conan_develop/src/linux/common/polling/utils.cpp:44
#21 0x00007ffff6f7c4ef in Common::Polling::Poll(std::function<bool ()> const&, std::function<int ()> const&, std::chrono::duration<long, std::ratio<1l, 1000l> > const&, std::function<bool ()> const&, short) (kill_functor=..., fd_getter=..., listen_interval=std::chrono::duration = { 250ms }, callback=..., fd_event=1) at /home/jenkins/workspace/common_conan_develop/src/linux/common/polling/utils.cpp:57
#22 0x00007ffff6fae910 in Common::Time::Timer::timer_poll (this=0x555556d4b990) at /home/jenkins/workspace/common_conan_develop/src/linux/common/time/timer.cpp:31
#23 0x00007ffff6fb2a5a in std::__invoke_impl<void, void (Common::Time::Timer::*)(), Common::Time::Timer*> (
    __f=@0x555556d58340: (void (Common::Time::Timer::*)(Common::Time::Timer * const)) 0x7ffff6fae82e <Common::Time::Timer::timer_poll()>, __t=@0x555556d58338: 0x555556d4b990)
    at /usr/include/c++/10/bits/invoke.h:73
#24 0x00007ffff6fb292d in std::__invoke<void (Common::Time::Timer::*)(), Common::Time::Timer*> (
    __fn=@0x555556d58340: (void (Common::Time::Timer::*)(Common::Time::Timer * const)) 0x7ffff6fae82e <Common::Time::Timer::timer_poll()>) at /usr/include/c++/10/bits/invoke.h:95
#25 0x00007ffff6fb2828 in std::thread::_Invoker<std::tuple<void (Common::Time::Timer::*)(), Common::Time::Timer*> >::_M_invoke<0ul, 1ul> (this=0x555556d58338) at /usr/include/c++/10/thread:264
#26 0x00007ffff6fb27a7 in std::thread::_Invoker<std::tuple<void (Common::Time::Timer::*)(), Common::Time::Timer*> >::operator() (this=0x555556d58338) at /usr/include/c++/10/thread:271
#27 0x00007ffff6fb2775 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (Common::Time::Timer::*)(), Common::Time::Timer*> > >::_M_run (this=0x555556d58330)
    at /usr/include/c++/10/thread:215
#28 0x00007ffff5f4a793 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#29 0x00007ffff7ed7609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#30 0x00007ffff5c1d353 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

I'm using the libraries from Conan:

        "uwebsockets/20.45.0",
        "usockets/0.8.6",

[ddevienne]

uWS is single-threaded, and uses thread_local variables. So while it can be used in a multi-threaded environment, you have to be careful when crossing thread boundaries. I don't recall whether defer() can be used safely across-threads. But given you crash, maybe not? FWIW.

[damirbarr]

I was using the defer() function so far for exactly this reason and it worked. For some reason now it sometimes crashes.
The defer function uses the loop's thread, and this is why I'm using it.
My software is multithreaded and I think there's maybe a bug here.

Reference from this repo:

    /* Defer this callback on Loop's thread of execution */
    void defer(MoveOnlyFunction<void()> &&cb) {
        LoopData *loopData = (LoopData *) us_loop_ext((us_loop_t *) this);

        //if (std::thread::get_id() == ) // todo: add fast path for same thread id
        loopData->deferMutex.lock();
        loopData->deferQueues[loopData->currentDeferQueue].emplace_back(std::move(cb));
        loopData->deferMutex.unlock();

        us_wakeup_loop((us_loop_t *) this);
    }

[uNetworkingAB]

defer is thread safe. But you need to check your composition with ThreadSanitizer or AddressSanitizer. The stacktrace is very weird and most likely not showing the real problem

[damirbarr]

Any suggestions other than TSAN and ASAN?