Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

logcabin abort after running out of file descriptors #198

Open
nhardt opened this issue Nov 3, 2015 · 3 comments
Open

logcabin abort after running out of file descriptors #198

nhardt opened this issue Nov 3, 2015 · 3 comments
Labels
bug

Comments

@nhardt
Copy link
Contributor

nhardt commented Nov 3, 2015

Scale has a test that will repeatedly block network traffic for 65 seconds at a time. Under these circumstances, logcabind can run out file descriptors and abort. During this time, there are an average number of attempts to read/write. Haven't yet managed to repro the issue with easy steps or in a unit test, and it's possible this is working by design, but just thought I'd post in case it triggers any ideas of which direction might need some exploration.

Stacks:
(gdb) thread apply all bt

Thread 23 (Thread 0x7ffcf8dfa700 (LWP 12967)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1aae6e8, lockGuard=...) at build/Core/ConditionVariable.cc:84
#2  0x0000000000550b15 in LogCabin::RPC::ThreadDispatchService::workerMain (this=0x1aae680) at build/RPC/ThreadDispatchService.cc:96
#3  0x0000000000554a05 in std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>::operator() (this=0x1aa6398, __object=0x1aae680) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
#4  0x000000000055497a in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aa6398, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
#5  0x000000000055491d in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::operator()<>(void) (this=0x1aa6398) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
#6  0x000000000055485e in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)> >::_M_run(void) (this=0x1aa6380) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
#7  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
#8  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00007ffd0290e8fd in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7ffd01c0e700 (LWP 12959)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1aa8290, lockGuard=...) at build/Core/ConditionVariable.cc:84
#2  0x0000000000560c57 in LogCabin::Core::ConditionVariable::wait (this=0x1aa8290, lockGuard=...) at build/Core/ConditionVariable.cc:100
#3  0x000000000044156e in LogCabin::Server::RaftConsensus::leaderDiskThreadMain (this=0x1aa8190) at build/Server/RaftConsensus.cc:2044
#4  0x000000000045a23f in std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>::operator() (this=0x1aa9ec8, __object=0x1aa8190) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
#5  0x000000000045a0f6 in std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>(LogCabin::Server::RaftConsensus*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aa9ec8, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
#6  0x000000000045a059 in std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>(LogCabin::Server::RaftConsensus*)>::operator()<>(void) (this=0x1aa9ec8) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
#7  0x0000000000459df6 in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>(LogCabin::Server::RaftConsensus*)> >::_M_run(void) (this=0x1aa9eb0) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
#8  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
#9  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
#10 0x00007ffd0290e8fd in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7ffcf3fff700 (LWP 12964)):
#0  0x00007ffd03fa398e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000560e86 in LogCabin::Core::ConditionVariable::wait_until (this=0x1ab3660, lockGuard=..., abs_time=...) at build/Core/ConditionVariable.cc:127
#2  0x000000000044ce77 in LogCabin::Core::ConditionVariable::wait_until<LogCabin::Core::Time::CSteadyClock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x1ab3660, lockGuard=..., abs_time=...) at ./Core/ConditionVariable.h:105
#3  0x0000000000463c87 in LogCabin::Server::StateMachine::snapshotThreadMain (this=0x1ab3570) at build/Server/StateMachine.cc:647
#4  0x000000000046c015 in std::_Mem_fn<void (LogCabin::Server::StateMachine::*)()>::operator() (this=0x1aa7258, __object=0x1ab3570) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
#5  0x000000000046bf8a in std::_Bind<std::_Mem_fn<void (LogCabin::Server::StateMachine::*)()>(LogCabin::Server::StateMachine*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aa7258, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
#6  0x000000000046bf2d in std::_Bind<std::_Mem_fn<void (LogCabin::Server::StateMachine::*)()>(LogCabin::Server::StateMachine*)>::operator()<>(void) (this=0x1aa7258) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
#7  0x000000000046be6e in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::Server::StateMachine::*)()>(LogCabin::Server::StateMachine*)> >::_M_run(void) (this=0x1aa7240) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
#8  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
#9  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
#10 0x00007ffd0290e8fd in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7ffd0120d700 (LWP 12960)):
#0  0x00007ffd03fa398e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000560e86 in LogCabin::Core::ConditionVariable::wait_until (this=0x1aa8290, lockGuard=..., abs_time=...) at build/Core/ConditionVariable.cc:127
#2  0x000000000044ce77 in LogCabin::Core::ConditionVariable::wait_until<LogCabin::Core::Time::CSteadyClock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x1aa8290, lockGuard=..., abs_time=...) at ./Core/ConditionVariable.h:105
#3  0x00000000004416ba in LogCabin::Server::RaftConsensus::timerThreadMain (this=0x1aa8190) at build/Server/RaftConsensus.cc:2056
#4  0x000000000045a23f in std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>::operator() (this=0x1aa6ff8, __object=0x1aa8190) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
#5  0x000000000045a0f6 in std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>(LogCabin::Server::RaftConsensus*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aa6ff8, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
#6  0x000000000045a059 in std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>(LogCabin::Server::RaftConsensus*)>::operator()<>(void) (this=0x1aa6ff8) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
#7  0x0000000000459df6 in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>(LogCabin::Server::RaftConsensus*)> >::_M_run(void) (this=0x1aa6fe0) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
#8  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
#9  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
#10 0x00007ffd0290e8fd in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7ffcf17fb700 (LWP 14482)):
#0  0x00007ffd0290eef3 in epoll_wait () from /lib64/libc.so.6
#1  0x00000000005556f0 in LogCabin::Event::Loop::runForever (this=0x7ffcf17fa350) at build/Event/Loop.cc:154
#2  0x0000000000540e1c in LogCabin::RPC::ClientSession::ClientSession (this=0x7ffcbc000910, eventLoop=..., address=..., maxMessageLength=1049600, timeout=..., config=...) at build/RPC/ClientSession.cc:301
#3  0x000000000054139c in LogCabin::RPC::ClientSession::makeSession (eventLoop=..., address=..., maxMessageLength=1049600, timeout=..., config=...) at build/RPC/ClientSession.cc:346
#4  0x00000000004c2525 in LogCabin::Client::SessionManager::createSession (this=0x1aa8230, address=..., timeout=..., clusterUUID=0x7fff826cfa30, serverId=0x7ffcf17faa00) at build/Client/SessionManager.cc:47
#5  0x0000000000438194 in LogCabin::Server::RaftConsensusInternal::Peer::getSession (this=0x7ffcc8012750, lockGuard=...) at build/Server/RaftConsensus.cc:339
#6  0x00000000004379f3 in LogCabin::Server::RaftConsensusInternal::Peer::callRPC (this=0x7ffcc8012750, opCode=LogCabin::Protocol::Raft::REQUEST_VOTE, request=..., response=..., lockGuard=...) at build/Server/RaftConsensus.cc:275
#7  0x0000000000444f5e in LogCabin::Server::RaftConsensus::requestVote (this=0x1aa8190, lockGuard=..., peer=...) at build/Server/RaftConsensus.cc:2769
#8  0x0000000000441898 in LogCabin::Server::RaftConsensus::peerThreadMain (this=0x1aa8190, peer=std::shared_ptr (count 5) 0x7ffcc8012750) at build/Server/RaftConsensus.cc:2086
#9  0x000000000045a309 in std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)(std::shared_ptr<LogCabin::Server::RaftConsensusInternal::Peer>)>::operator() (this=0x7ffcc8006758, __object=0x1aa8190, __args#0=std::shared_ptr (count 5) 0x7ffcc8012750) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
#10 0x000000000045a17f in std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)(std::shared_ptr<LogCabin::Server::RaftConsensusInternal::Peer>)>(LogCabin::Server::RaftConsensus*, std::shared_ptr<LogCabin::Server::RaftConsensusInternal::Peer>)>::__call<, 0, 1>(const std::tuple<> &, std::_Index_tuple<0, 1>) (this=0x7ffcc8006758, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
#11 0x000000000045a08b in std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)(std::shared_ptr<LogCabin::Server::RaftConsensusInternal::Peer>)>(LogCabin::Server::RaftConsensus*, std::shared_ptr<LogCabin::Server::RaftConsensusInternal::Peer>)>::operator()<>(void) (this=0x7ffcc8006758) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
#12 0x0000000000459f08 in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)(std::shared_ptr<LogCabin::Server::RaftConsensusInternal::Peer>)>(LogCabin::Server::RaftConsensus*, std::shared_ptr<LogCabin::Server::RaftConsensusInternal::Peer>)> >::_M_run(void) (this=0x7ffcc8006740) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
#13 0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
#14 0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
#15 0x00007ffd0290e8fd in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7ffcf97fb700 (LWP 12966)):
#0  0x00007ffd03fa398e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000560e86 in LogCabin::Core::ConditionVariable::wait_until (this=0x7fff826cf6c0, lockGuard=..., abs_time=...) at build/Core/ConditionVariable.cc:127
#2  0x000000000044ce77 in LogCabin::Core::ConditionVariable::wait_until<LogCabin::Core::Time::CSteadyClock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fff826cf6c0, lockGuard=..., abs_time=...) at ./Core/ConditionVariable.h:105
#3  0x000000000045e63a in LogCabin::Server::ServerStats::statsDumperMain (this=0x7fff826cf670) at build/Server/ServerStats.cc:196
#4  0x00000000004600c3 in std::_Mem_fn<void (LogCabin::Server::ServerStats::*)()>::operator() (this=0x1aaeb38, __object=0x7fff826cf670) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
#5  0x0000000000460038 in std::_Bind<std::_Mem_fn<void (LogCabin::Server::ServerStats::*)()>(LogCabin::Server::ServerStats*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aaeb38, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
#6  0x000000000045ffdb in std::_Bind<std::_Mem_fn<void (LogCabin::Server::ServerStats::*)()>(LogCabin::Server::ServerStats*)>::operator()<>(void) (this=0x1aaeb38) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
#7  0x000000000045ff1c in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::Server::ServerStats::*)()>(LogCabin::Server::ServerStats*)> >::_M_run(void) (this=0x1aaeb20) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
#8  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
#9  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
#10 0x00007ffd0290e8fd in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7ffcbbfff700 (LWP 18089)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1aae6e8, lockGuard=...) at build/Core/ConditionVariable.cc:84
#2  0x0000000000550b15 in LogCabin::RPC::ThreadDispatchService::workerMain (this=0x1aae680) at build/RPC/ThreadDispatchService.cc:96
#3  0x0000000000554a05 in std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>::operator() (this=0x1aaf668, __object=0x1aae680) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
#4  0x000000000055497a in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aaf668, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
#5  0x000000000055491d in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::operator()<>(void) (this=0x1aaf668) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
#6  0x000000000055485e in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)> >::_M_run(void) (this=0x1aaf650) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
#7  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
#8  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00007ffd0290e8fd in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7ffcf35fe700 (LWP 12968)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1aae6e8, lockGuard=...) at build/Core/ConditionVariable.cc:84
#2  0x0000000000550b15 in LogCabin::RPC::ThreadDispatchService::workerMain (this=0x1aae680) at build/RPC/ThreadDispatchService.cc:96
#3  0x0000000000554a05 in std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>::operator() (this=0x1aa6218, __object=0x1aae680) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
#4  0x000000000055497a in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aa6218, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
#5  0x000000000055491d in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::operator()<>(void) (this=0x1aa6218) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
#6  0x000000000055485e in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)> >::_M_run(void) (this=0x1aa6200) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
#7  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
#8  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00007ffd0290e8fd in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7ffcf2bfd700 (LWP 12969)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1aae6e8, lockGuard=...) at build/Core/ConditionVariable.cc:84
#2  0x0000000000550b15 in LogCabin::RPC::ThreadDispatchService::workerMain (this=0x1aae680) at build/RPC/ThreadDispatchService.cc:96
#3  0x0000000000554a05 in std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>::operator() (this=0x1aa6158, __object=0x1aae680) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
#4  0x000000000055497a in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aa6158, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
#5  0x000000000055491d in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::operator()<>(void) (this=0x1aa6158) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
#6  0x000000000055485e in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)> >::_M_run(void) (this=0x1aa6140) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
#7  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
#8  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00007ffd0290e8fd in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7ffcf21fc700 (LWP 14481)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1ab0ab8, lockGuard=...) at build/Core/ConditionVariable.cc:84
#2  0x0000000000550b15 in LogCabin::RPC::ThreadDispatchService::workerMain (this=0x1ab0a50) at build/RPC/ThreadDispatchService.cc:96
#3  0x0000000000554a05 in std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>::operator() (this=0x1ab40b8, __object=0x1ab0a50) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
#4  0x000000000055497a in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1ab40b8, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
#5  0x000000000055491d in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::operator()<>(void) (this=0x1ab40b8) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
#6  0x000000000055485e in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)> >::_M_run(void) (this=0x1ab40a0) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
#7  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
#8  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00007ffd0290e8fd in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7ffcb97fb700 (LWP 25063)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1aae6e8, lockGuard=...) at build/Core/ConditionVariable.cc:84
#2  0x0000000000550b15 in LogCabin::RPC::ThreadDispatchService::workerMain (this=0x1aae680) at build/RPC/ThreadDispatchService.cc:96
#3  0x0000000000554a05 in std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>::operator() (this=0x1aaecf8, __object=0x1aae680) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
#4  0x000000000055497a in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aaecf8, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
#5  0x000000000055491d in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::operator()<>(void) (this=0x1aaecf8) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
#6  0x000000000055485e in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)> >::_M_run(void) (this=0x1aaece0) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
#7  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
#8  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00007ffd0290e8fd in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7ffd0260f700 (LWP 12958)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1aaef40, lockGuard=...) at build/Core/ConditionVariable.cc:84
#2  0x0000000000560c57 in LogCabin::Core::ConditionVariable::wait (this=0x1aaef40, lockGuard=...) at build/Core/ConditionVariable.cc:100
#3  0x00000000004802db in LogCabin::Storage::SegmentedLog::PreparedSegments::waitForDemand (this=0x1aaeef0) at build/Storage/SegmentedLog.cc:138
#4  0x0000000000486e42 in LogCabin::Storage::SegmentedLog::segmentPreparerMain (this=0x1aaee00) at build/Storage/SegmentedLog.cc:1350
#5  0x0000000000495231 in std::_Mem_fn<void (LogCabin::Storage::SegmentedLog::*)()>::operator() (this=0x1aaaf78, __object=0x1aaee00) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
#6  0x00000000004951a6 in std::_Bind<std::_Mem_fn<void (LogCabin::Storage::SegmentedLog::*)()>(LogCabin::Storage::SegmentedLog*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aaaf78, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
#7  0x0000000000495149 in std::_Bind<std::_Mem_fn<void (LogCabin::Storage::SegmentedLog::*)()>(LogCabin::Storage::SegmentedLog*)>::operator()<>(void) (this=0x1aaaf78) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
#8  0x000000000049508a in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::Storage::SegmentedLog::*)()>(LogCabin::Storage::SegmentedLog*)> >::_M_run(void) (this=0x1aaaf60) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
#9  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
#10 0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
#11 0x00007ffd0290e8fd in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7ffcb8dfa700 (LWP 17183)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1ab0ab8, lockGuard=...) at build/Core/ConditionVariable.cc:84
#2  0x0000000000550b15 in LogCabin::RPC::ThreadDispatchService::workerMain (this=0x1ab0a50) at build/RPC/ThreadDispatchService.cc:96
#3  0x0000000000554a05 in std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>::operator() (this=0x1acfd78, __object=0x1ab0a50) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
#4  0x000000000055497a in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1acfd78, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
#5  0x000000000055491d in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::operator()<>(void) (this=0x1acfd78) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
#6  0x000000000055485e in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)> >::_M_run(void) (this=0x1acfd60) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
#7  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
#8  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00007ffd0290e8fd in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7ffcbb5fe700 (LWP 18090)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1aae6e8, lockGuard=...) at build/Core/ConditionVariable.cc:84
#2  0x0000000000550b15 in LogCabin::RPC::ThreadDispatchService::workerMain (this=0x1aae680) at build/RPC/ThreadDispatchService.cc:96
#3  0x0000000000554a05 in std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>::operator() (this=0x1aa5a68, __object=0x1aae680) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
#4  0x000000000055497a in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aa5a68, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
#5  0x000000000055491d in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::operator()<>(void) (this=0x1aa5a68) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
#6  0x000000000055485e in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)> >::_M_run(void) (this=0x1aa5a50) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
#7  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
#8  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00007ffd0290e8fd in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7ffcbabfd700 (LWP 19137)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1aae6e8, lockGuard=...) at build/Core/ConditionVariable.cc:84
#2  0x0000000000550b15 in LogCabin::RPC::ThreadDispatchService::workerMain (this=0x1aae680) at build/RPC/ThreadDispatchService.cc:96
#3  0x0000000000554a05 in std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>::operator() (this=0x1aaccb8, __object=0x1aae680) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
#4  0x000000000055497a in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aaccb8, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
#5  0x000000000055491d in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::operator()<>(void) (this=0x1aaccb8) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
#6  0x000000000055485e in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)> >::_M_run(void) (this=0x1aacca0) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
#7  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
#8  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00007ffd0290e8fd in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7ffca3fff700 (LWP 26705)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1aae6e8, lockGuard=...) at build/Core/ConditionVariable.cc:84
#2  0x0000000000550b15 in LogCabin::RPC::ThreadDispatchService::workerMain (this=0x1aae680) at build/RPC/ThreadDispatchService.cc:96
#3  0x0000000000554a05 in std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>::operator() (this=0x1af4848, __object=0x1aae680) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
#4  0x000000000055497a in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1af4848, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
#5  0x000000000055491d in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::operator()<>(void) (this=0x1af4848) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
#6  0x000000000055485e in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)> >::_M_run(void) (this=0x1af4830) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
#7  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
#8  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00007ffd0290e8fd in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7ffcfb5fe700 (LWP 12962)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1aa8290, lockGuard=...) at build/Core/ConditionVariable.cc:84
#2  0x0000000000560c57 in LogCabin::Core::ConditionVariable::wait (this=0x1aa8290, lockGuard=...) at build/Core/ConditionVariable.cc:100
#3  0x0000000000441c28 in LogCabin::Server::RaftConsensus::stepDownThreadMain (this=0x1aa8190) at build/Server/RaftConsensus.cc:2135
#4  0x000000000045a23f in std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>::operator() (this=0x1aa7188, __object=0x1aa8190) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
#5  0x000000000045a0f6 in std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>(LogCabin::Server::RaftConsensus*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aa7188, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
#6  0x000000000045a059 in std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>(LogCabin::Server::RaftConsensus*)>::operator()<>(void) (this=0x1aa7188) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
#7  0x0000000000459df6 in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>(LogCabin::Server::RaftConsensus*)> >::_M_run(void) (this=0x1aa7170) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
#8  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
#9  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
#10 0x00007ffd0290e8fd in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7ffcba1fc700 (LWP 19138)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1aae6e8, lockGuard=...) at build/Core/ConditionVariable.cc:84
#2  0x0000000000550b15 in LogCabin::RPC::ThreadDispatchService::workerMain (this=0x1aae680) at build/RPC/ThreadDispatchService.cc:96
#3  0x0000000000554a05 in std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>::operator() (this=0x1aacbf8, __object=0x1aae680) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
#4  0x000000000055497a in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aacbf8, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
#5  0x000000000055491d in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::operator()<>(void) (this=0x1aacbf8) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
#6  0x000000000055485e in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)> >::_M_run(void) (this=0x1aacbe0) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
#7  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
#8  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
#9  0x00007ffd0290e8fd in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7ffcf0dfa700 (LWP 14483)):
#0  0x00007ffd0290eef3 in epoll_wait () from /lib64/libc.so.6
#1  0x00000000005556f0 in LogCabin::Event::Loop::runForever (this=0x7ffcf0df9350) at build/Event/Loop.cc:154
#2  0x0000000000540e1c in LogCabin::RPC::ClientSession::ClientSession (this=0x7ffcc0000b30, eventLoop=..., address=..., maxMessageLength=1049600, timeout=..., config=...) at build/RPC/ClientSession.cc:301
#3  0x000000000054139c in LogCabin::RPC::ClientSession::makeSession (eventLoop=..., address=..., maxMessageLength=1049600, timeout=..., config=...) at build/RPC/ClientSession.cc:346
#4  0x00000000004c2525 in LogCabin::Client::SessionManager::createSession (this=0x1aa8230, address=..., timeout=..., clusterUUID=0x7fff826cfa30, serverId=0x7ffcf0df9a00) at build/Client/SessionManager.cc:47
#5  0x0000000000438194 in LogCabin::Server::RaftConsensusInternal::Peer::getSession (this=0x7ffcc8004d00, lockGuard=...) at build/Server/RaftConsensus.cc:339
#6  0x00000000004379f3 in LogCabin::Server::RaftConsensusInternal::Peer::callRPC (this=0x7ffcc8004d00, opCode=LogCabin::Protocol::Raft::REQUEST_VOTE, request=..., response=..., lockGuard=...) at build/Server/RaftConsensus.cc:275
#7  0x0000000000444f5e in LogCabin::Server::RaftConsensus::requestVote (this=0x1aa8190, lockGuard=..., peer=...) at build/Server/RaftConsensus.cc:2769
#8  0x0000000000441898 in LogCabin::Server::RaftConsensus::peerThreadMain (this=0x1aa8190, peer=std::shared_ptr (count 5) 0x7ffcc8004d00) at build/Server/RaftConsensus.cc:2086
#9  0x000000000045a309 in std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)(std::shared_ptr<LogCabin::Server::RaftConsensusInternal::Peer>)>::operator() (this=0x7ffcc8007c88, __object=0x1aa8190, __args#0=std::shared_ptr (count 5) 0x7ffcc8004d00) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
#10 0x000000000045a17f in std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)(std::shared_ptr<LogCabin::Server::RaftConsensusInternal::Peer>)>(LogCabin::Server::RaftConsensus*, std::shared_ptr<LogCabin::Server::RaftConsensusInternal::Peer>)>::__call<, 0, 1>(const std::tuple<> &, std::_Index_tuple<0, 1>) (this=0x7ffcc8007c88, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
#11 0x000000000045a08b in std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)(std::shared_ptr<LogCabin::Server::RaftConsensusInternal::Peer>)>(LogCabin::Server::RaftConsensus*, std::shared_ptr<LogCabin::Server::RaftConsensusInternal::Peer>)>::operator()<>(void) (this=0x7ffcc8007c88) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
#12 0x0000000000459f08 in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)(std::shared_ptr<LogCabin::Server::RaftConsensusInternal::Peer>)>(LogCabin::Server::RaftConsensus*, std::shared_ptr<LogCabin::Server::RaftConsensusInternal::Peer>)> >::_M_run(void) (this=0x7ffcc8007c70) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
#13 0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
#14 0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
#15 0x00007ffd0290e8fd in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7ffcfa1fc700 (LWP 12965)):
#0  0x00007ffd03fa398e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000560e86 in LogCabin::Core::ConditionVariable::wait_until (this=0x1ab36c0, lockGuard=..., abs_time=...) at build/Core/ConditionVariable.cc:127
#2  0x000000000044ce77 in LogCabin::Core::ConditionVariable::wait_until<LogCabin::Core::Time::CSteadyClock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x1ab36c0, lockGuard=..., abs_time=...) at ./Core/ConditionVariable.h:105
#3  0x00000000004640cf in LogCabin::Server::StateMachine::snapshotWatchdogThreadMain (this=0x1ab3570) at build/Server/StateMachine.cc:714
#4  0x000000000046c015 in std::_Mem_fn<void (LogCabin::Server::StateMachine::*)()>::operator() (this=0x1ab19f8, __object=0x1ab3570) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
#5  0x000000000046bf8a in std::_Bind<std::_Mem_fn<void (LogCabin::Server::StateMachine::*)()>(LogCabin::Server::StateMachine*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1ab19f8, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
#6  0x000000000046bf2d in std::_Bind<std::_Mem_fn<void (LogCabin::Server::StateMachine::*)()>(LogCabin::Server::StateMachine*)>::operator()<>(void) (this=0x1ab19f8) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
#7  0x000000000046be6e in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::Server::StateMachine::*)()>(LogCabin::Server::StateMachine*)> >::_M_run(void) (this=0x1ab19e0) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
#8  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
#9  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
#10 0x00007ffd0290e8fd in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7ffcfabfd700 (LWP 12963)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1aa8290, lockGuard=...) at build/Core/ConditionVariable.cc:84
#2  0x0000000000560c57 in LogCabin::Core::ConditionVariable::wait (this=0x1aa8290, lockGuard=...) at build/Core/ConditionVariable.cc:100
#3  0x000000000043d450 in LogCabin::Server::RaftConsensus::getNextEntry (this=0x1aa8190, lastIndex=16689) at build/Server/RaftConsensus.cc:1242
#4  0x00000000004626e9 in LogCabin::Server::StateMachine::applyThreadMain (this=0x1ab3570) at build/Server/StateMachine.cc:391
#5  0x000000000046c015 in std::_Mem_fn<void (LogCabin::Server::StateMachine::*)()>::operator() (this=0x1aa77f8, __object=0x1ab3570) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
#6  0x000000000046bf8a in std::_Bind<std::_Mem_fn<void (LogCabin::Server::StateMachine::*)()>(LogCabin::Server::StateMachine*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aa77f8, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
#7  0x000000000046bf2d in std::_Bind<std::_Mem_fn<void (LogCabin::Server::StateMachine::*)()>(LogCabin::Server::StateMachine*)>::operator()<>(void) (this=0x1aa77f8) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
#8  0x000000000046be6e in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::Server::StateMachine::*)()>(LogCabin::Server::StateMachine*)> >::_M_run(void) (this=0x1aa77e0) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
#9  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
#10 0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
#11 0x00007ffd0290e8fd in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7ffcfbfff700 (LWP 12961)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1aa8290, lockGuard=...) at build/Core/ConditionVariable.cc:84
#2  0x0000000000560c57 in LogCabin::Core::ConditionVariable::wait (this=0x1aa8290, lockGuard=...) at build/Core/ConditionVariable.cc:100
#3  0x0000000000441214 in LogCabin::Server::RaftConsensus::stateMachineUpdaterThreadMain (this=0x1aa8190) at build/Server/RaftConsensus.cc:2009
#4  0x000000000045a23f in std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>::operator() (this=0x1aa70b8, __object=0x1aa8190) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
#5  0x000000000045a0f6 in std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>(LogCabin::Server::RaftConsensus*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aa70b8, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
#6  0x000000000045a059 in std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>(LogCabin::Server::RaftConsensus*)>::operator()<>(void) (this=0x1aa70b8) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
#7  0x0000000000459df6 in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>(LogCabin::Server::RaftConsensus*)> >::_M_run(void) (this=0x1aa70a0) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
#8  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
#9  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
#10 0x00007ffd0290e8fd in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7ffd043c4720 (LWP 12956)):
#0  0x00007ffd02858625 in raise () from /lib64/libc.so.6
#1  0x00007ffd02859e05 in abort () from /lib64/libc.so.6
#2  0x0000000000544a45 in LogCabin::RPC::(anonymous namespace)::dupOrPanic (oldfd=1023) at build/RPC/MessageSocket.cc:43
#3  0x00000000005450d5 in LogCabin::RPC::MessageSocket::MessageSocket (this=0x1b15e00, handler=..., eventLoop=..., fd=1023, maxMessageLength=1049600) at build/RPC/MessageSocket.cc:186
#4  0x000000000054885b in LogCabin::RPC::OpaqueServer::SocketWithHandler::SocketWithHandler (this=0x1b15de0, server=0x1aab7e8, fd=1023) at build/RPC/OpaqueServer.cc:115
#5  0x0000000000548789 in LogCabin::RPC::OpaqueServer::SocketWithHandler::make (server=0x1aab7e8, fd=1023) at build/RPC/OpaqueServer.cc:106
#6  0x0000000000548a05 in LogCabin::RPC::OpaqueServer::BoundListener::handleFileEvent (this=0x1aa8470, events=1) at build/RPC/OpaqueServer.cc:146
#7  0x00000000005557c7 in LogCabin::Event::Loop::runForever (this=0x7fff826cf788) at build/Event/Loop.cc:162
#8  0x000000000043466a in LogCabin::Server::Globals::run (this=0x7fff826cf630) at build/Server/Globals.cc:197
#9  0x000000000042db04 in main (argc=8, argv=0x7fff826d01a8) at build/Server/Main.cc:352


(gdb) thread apply all bt full

Thread 23 (Thread 0x7ffcf8dfa700 (LWP 12967)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1aae6e8, lockGuard=...) at build/Core/ConditionVariable.cc:84
        mutex = 0x1aae6a0
        r = 0
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x591c37)>
#2  0x0000000000550b15 in LogCabin::RPC::ThreadDispatchService::workerMain (this=0x1aae680) at build/RPC/ThreadDispatchService.cc:96
        lockGuard = {_M_device = 0x1aae6a0, _M_owns = true}
        rpc = {opaqueRPC = {request = {data = 0x0, length = 0, deleter = 0}, response = {data = 0x0, length = 0, deleter = 0}, socket = std::weak_ptr (empty) 0x0, messageId = 18446744073709551615, responseTarget = 0x0}, active = false, service = 0, serviceSpecificErrorVersion = 0 '\000', opCode = 0}
#3  0x0000000000554a05 in std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>::operator() (this=0x1aa6398, __object=0x1aae680) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
No locals.
#4  0x000000000055497a in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aa6398, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
No locals.
#5  0x000000000055491d in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::operator()<>(void) (this=0x1aa6398) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
No locals.
#6  0x000000000055485e in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)> >::_M_run(void) (this=0x1aa6380) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
No locals.
#7  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#8  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#9  0x00007ffd0290e8fd in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 22 (Thread 0x7ffd01c0e700 (LWP 12959)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1aa8290, lockGuard=...) at build/Core/ConditionVariable.cc:84
        mutex = 0x1aa8248
        r = 0
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x591c37)>
#2  0x0000000000560c57 in LogCabin::Core::ConditionVariable::wait (this=0x1aa8290, lockGuard=...) at build/Core/ConditionVariable.cc:100
        mutex = @0x1aa8248
        __PRETTY_FUNCTION__ = <error reading variable __PRETTY_FUNCTION__ (Cannot access memory at address 0x591be0)>
        stdLockGuard = {_M_device = 0x1aa8248, _M_owns = true}
#3  0x000000000044156e in LogCabin::Server::RaftConsensus::leaderDiskThreadMain (this=0x1aa8190) at build/Server/RaftConsensus.cc:2044
        lockGuard = {_M_device = 0x0, _M_owns = false}
#4  0x000000000045a23f in std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>::operator() (this=0x1aa9ec8, __object=0x1aa8190) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
No locals.
#5  0x000000000045a0f6 in std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>(LogCabin::Server::RaftConsensus*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aa9ec8, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
No locals.
#6  0x000000000045a059 in std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>(LogCabin::Server::RaftConsensus*)>::operator()<>(void) (this=0x1aa9ec8) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
No locals.
#7  0x0000000000459df6 in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>(LogCabin::Server::RaftConsensus*)> >::_M_run(void) (this=0x1aa9eb0) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
No locals.
#8  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#9  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#10 0x00007ffd0290e8fd in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 21 (Thread 0x7ffcf3fff700 (LWP 12964)):
#0  0x00007ffd03fa398e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000560e86 in LogCabin::Core::ConditionVariable::wait_until (this=0x1ab3660, lockGuard=..., abs_time=...) at build/Core/ConditionVariable.cc:127
        wake = {__d = {__r = 33435465625946}}
        mutex = 0x1ab35b8
        now = {__d = {__r = 29835465625946}}
        wakespec = {tv_sec = 33435, tv_nsec = 465625946}
        r = 0
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x591bcc)>
#2  0x000000000044ce77 in LogCabin::Core::ConditionVariable::wait_until<LogCabin::Core::Time::CSteadyClock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x1ab3660, lockGuard=..., abs_time=...) at ./Core/ConditionVariable.h:105
        __PRETTY_FUNCTION__ = <error reading variable __PRETTY_FUNCTION__ (Cannot access memory at address 0x57c6a0)>
        mutex = @0x1ab35b8
        stdLockGuard = {_M_device = 0x1ab35b8, _M_owns = true}
#3  0x0000000000463c87 in LogCabin::Server::StateMachine::snapshotThreadMain (this=0x1ab3570) at build/Server/StateMachine.cc:647
        inhibited = false
        waitUntil = {__d = {__r = 9223372036854775807}}
        lockGuard = {_M_device = 0x0, _M_owns = false}
        wasInhibited = false
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x57feb0)>
#4  0x000000000046c015 in std::_Mem_fn<void (LogCabin::Server::StateMachine::*)()>::operator() (this=0x1aa7258, __object=0x1ab3570) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
No locals.
#5  0x000000000046bf8a in std::_Bind<std::_Mem_fn<void (LogCabin::Server::StateMachine::*)()>(LogCabin::Server::StateMachine*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aa7258, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
No locals.
#6  0x000000000046bf2d in std::_Bind<std::_Mem_fn<void (LogCabin::Server::StateMachine::*)()>(LogCabin::Server::StateMachine*)>::operator()<>(void) (this=0x1aa7258) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
No locals.
#7  0x000000000046be6e in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::Server::StateMachine::*)()>(LogCabin::Server::StateMachine*)> >::_M_run(void) (this=0x1aa7240) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
No locals.
#8  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#9  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#10 0x00007ffd0290e8fd in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 20 (Thread 0x7ffd0120d700 (LWP 12960)):
#0  0x00007ffd03fa398e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000560e86 in LogCabin::Core::ConditionVariable::wait_until (this=0x1aa8290, lockGuard=..., abs_time=...) at build/Core/ConditionVariable.cc:127
        wake = {__d = {__r = 31758046281036}}
        mutex = 0x1aa8248
        now = {__d = {__r = 31755374413331}}
        wakespec = {tv_sec = 31758, tv_nsec = 46281036}
        r = 0
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x591bcc)>
#2  0x000000000044ce77 in LogCabin::Core::ConditionVariable::wait_until<LogCabin::Core::Time::CSteadyClock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x1aa8290, lockGuard=..., abs_time=...) at ./Core/ConditionVariable.h:105
        __PRETTY_FUNCTION__ = <error reading variable __PRETTY_FUNCTION__ (Cannot access memory at address 0x57c6a0)>
        mutex = @0x1aa8248
        stdLockGuard = {_M_device = 0x1aa8248, _M_owns = true}
#3  0x00000000004416ba in LogCabin::Server::RaftConsensus::timerThreadMain (this=0x1aa8190) at build/Server/RaftConsensus.cc:2056
        lockGuard = {_M_device = 0x0, _M_owns = false}
#4  0x000000000045a23f in std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>::operator() (this=0x1aa6ff8, __object=0x1aa8190) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
No locals.
#5  0x000000000045a0f6 in std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>(LogCabin::Server::RaftConsensus*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aa6ff8, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
No locals.
#6  0x000000000045a059 in std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>(LogCabin::Server::RaftConsensus*)>::operator()<>(void) (this=0x1aa6ff8) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
No locals.
#7  0x0000000000459df6 in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>(LogCabin::Server::RaftConsensus*)> >::_M_run(void) (this=0x1aa6fe0) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
No locals.
#8  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#9  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#10 0x00007ffd0290e8fd in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 19 (Thread 0x7ffcf17fb700 (LWP 14482)):
#0  0x00007ffd0290eef3 in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
#1  0x00000000005556f0 in LogCabin::Event::Loop::runForever (this=0x7ffcf17fa350) at build/Event/Loop.cc:154
        extraLockGuard = {_M_device = 0x7ffcf17fa470, _M_owns = true}
        events = {{events = 109466694, data = {ptr = 0xf17fa57000001ce2, fd = 7394, u32 = 7394, u64 = 17401809385637813474}}}
        r = 31757
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x58f294)>
#2  0x0000000000540e1c in LogCabin::RPC::ClientSession::ClientSession (this=0x7ffcbc000910, eventLoop=..., address=..., maxMessageLength=1049600, timeout=..., config=...) at build/RPC/ClientSession.cc:301
        loop = {epollfd = 1004, breakTimer = {<LogCabin::Event::Timer> = {<LogCabin::Event::File> = {_vptr.File = 0x8272d0, fd = 1010, ownership = LogCabin::Event::File::CLOSE_ON_DESTROY}, <No data fields>}, <No data fields>}, shouldExit = false, mutex = {_M_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, runningThread = 15, numLocks = 0, numActiveLocks = 0, lockOwner = 0, safeToLock = {cv = {__data = {__lock = 0, __futex = 0, __total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0, __nwaiters = 1, __broadcast_seq = 0}, __size = '\000' <repeats 40 times>, "\001\000\000\000\000\000\000", __align = 0}, callback = {<std::_Maybe_unary_or_binary_function<void>> = {<No data fields>}, <std::_Function_base> = {static _M_max_size = 16, static _M_max_align = 8, _M_functor = {_M_unused = {_M_object = 0x7ffc00000005, _M_const_object = 0x7ffc00000005, _M_function_pointer = 0x7ffc00000005, _M_member_pointer = &virtual table offset 140720308486148, this adjustment 1}, _M_pod_data = "\005\000\000\000\374\177\000\000\001\000\000\000\000\000\000"}, _M_manager = 0}, _M_invoker = 0x70}, notificationCount = {<std::__atomic2::__atomic_base<unsigned long>> = {_M_i = 0}, <No data fields>}, lastWaitUntil = {__d = {__r = 0}}}, unlocked = {cv = {__data = {__lock = 0, __futex = 0, __total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0, __nwaiters = 1, __broadcast_seq = 0}, __size = '\000' <repeats 40 times>, "\001\000\000\000\000\000\000", __align = 0}, callback = {<std::_Maybe_unary_or_binary_function<void>> = {<No data fields>}, <std::_Function_base> = {static _M_max_size = 16, static _M_max_align = 8, _M_functor = {_M_unused = {_M_object = 0x40, _M_const_object = 0x40, _M_function_pointer = 0x40, _M_member_pointer = (void (std::_Undefined_class::*)(std::_Undefined_class *)) 0x40, this adjustment 140723462602784}, _M_pod_data = "@\000\000\000\000\000\000\000 \000\000\274\374\177\000"}, _M_manager = 0}, _M_invoker = 0x2}, notificationCount = {<std::__atomic2::__atomic_base<unsigned long>> = {_M_i = 0}, <No data fields>}, lastWaitUntil = {__d = {__r = 0}}}, extraMutexToSatisfyRaceDetector = {<No data fields>}, breakTimerMonitor = {<LogCabin::Event::File::Monitor> = {_vptr.Monitor = 0x827450, eventLoop = @0x7ffcf17fa350, mutex = {_M_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, file = 0x7ffcf17fa358}, <No data fields>}}
        fileNotifier = {<LogCabin::Event::File> = {_vptr.File = 0x81ded0, fd = 1009, ownership = LogCabin::Event::File::CALLER_CLOSES_FD}, eventLoop = @0x7ffcf17fa350, count = 0}
        timerNotifier = {<LogCabin::Event::Timer> = {<LogCabin::Event::File> = {_vptr.File = 0x826af0, fd = 1017, ownership = LogCabin::Event::File::CLOSE_ON_DESTROY}, <No data fields>}, eventLoop = @0x7ffcf17fa350}
        fileMonitor = {_vptr.Monitor = 0x827290, eventLoop = @0x7ffcf17fa350, mutex = {_M_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, file = 0x7ffcf17fa540}
        timerMonitor = {<LogCabin::Event::File::Monitor> = {_vptr.Monitor = 0x827450, eventLoop = @0x7ffcf17fa350, mutex = {_M_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, file = 0x7ffcf17fa570}, <No data fields>}
        fd = 1009
        waiting = true
        r = -1
#3  0x000000000054139c in LogCabin::RPC::ClientSession::makeSession (eventLoop=..., address=..., maxMessageLength=1049600, timeout=..., config=...) at build/RPC/ClientSession.cc:346
        session = std::shared_ptr (empty) 0x0
#4  0x00000000004c2525 in LogCabin::Client::SessionManager::createSession (this=0x1aa8230, address=..., timeout=..., clusterUUID=0x7fff826cfa30, serverId=0x7ffcf17faa00) at build/Client/SessionManager.cc:47
        session = std::shared_ptr (empty) 0x0
        rpc = {service = 42896, opCode = 61823, opaqueRPC = {mutex = {_M_mutex = {__data = {__lock = 4385355, __count = 0, __owner = -243292208, __nusers = 32764, __kind = 8577888, __spins = 0, __list = {__prev = 0x7ffcf17fa7b0, __next = 0x4305ba}}, __size = "K\352B\000\000\000\000\000Ч\177\361\374\177\000\000`\343\202\000\000\000\000\000\260\247\177\361\374\177\000\000\272\005C\000\000\000\000", __align = 4385355}}, session = std::shared_ptr (count 0) 0x82e4a0, responseToken = 140724360161232, status = 4393669, reply = {data = 0x7ffcf17fa820, length = 140724360161296, deleter = 0x7ffcf17fa850}, errorMessage = Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 558, in to_string
    return self.val['_M_dataplus']['_M_p'].lazy_string (length = len)
RuntimeError: Cannot access memory at address 0x566bcc
}}
        response = {<google::protobuf::Message> = {<No data fields>}, static kClusterUuidFieldNumber = 1, static kServerIdFieldNumber = 2, static kOkFieldNumber = 3, static kErrorFieldNumber = 4, _unknown_fields_ = {fields_ = 0x1aacff8}, cluster_uuid_ = 0x58d0da, server_id_ = 171804183902, error_ = 0x1aac510, ok_ = 208, _cached_size_ = 0, _has_bits_ = {8577888}, static default_instance_ = 0x1ab0b80}
        status = LogCabin::RPC::ClientRPC::OK
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x587c10)>
        request = {<google::protobuf::Message> = {<No data fields>}, static kClusterUuidFieldNumber = 1, static kServerIdFieldNumber = 2, _unknown_fields_ = {fields_ = 0x53c07a}, cluster_uuid_ = 0x7ffcc8012760, server_id_ = 140724360161616, _cached_size_ = -185583672, _has_bits_ = {7394}, static default_instance_ = 0x1ab1790}
        error = {<google::protobuf::Message> = {<No data fields>}, static UNKNOWN = LogCabin::Protocol::Client::Error_Code_UNKNOWN, static NOT_LEADER = LogCabin::Protocol::Client::Error_Code_NOT_LEADER, static Code_MIN = LogCabin::Protocol::Client::Error_Code_UNKNOWN, static Code_MAX = LogCabin::Protocol::Client::Error_Code_NOT_LEADER, static Code_ARRAYSIZE = 2, static kErrorCodeFieldNumber = 1, static kLeaderHintFieldNumber = 2, _unknown_fields_ = {fields_ = 0x1aacff8}, leader_hint_ = 0x3, error_code_ = 42948152, _cached_size_ = 30, _has_bits_ = {721799399}, static default_instance_ = 0x1ab1470}
#5  0x0000000000438194 in LogCabin::Server::RaftConsensusInternal::Peer::getSession (this=0x7ffcc8012750, lockGuard=...) at build/Server/RaftConsensus.cc:339
        timeout = {__d = {__r = 31761097570248}}
        unlockGuard = {guard = @0x7ffcf17fad30}
        target = {originalString = "192.168.8.33", hosts = std::vector of length 1, capacity 1 = {{first = "192.168.8.33", second = "5254"}}, storage = {ss_family = 2, __ss_align = 0, __ss_padding = '\000' <repeats 111 times>}, len = 16}
        peerId = {mutex = {_M_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, empty = false, value = 1}
#6  0x00000000004379f3 in LogCabin::Server::RaftConsensusInternal::Peer::callRPC (this=0x7ffcc8012750, opCode=LogCabin::Protocol::Raft::REQUEST_VOTE, request=..., response=..., lockGuard=...) at build/Server/RaftConsensus.cc:275
        unlockGuard = {guard = @0x1ce1caea411a}
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x57ce93)>
#7  0x0000000000444f5e in LogCabin::Server::RaftConsensus::requestVote (this=0x1aa8190, lockGuard=..., peer=...) at build/Server/RaftConsensus.cc:2769
        request = {<google::protobuf::Message> = {<No data fields>}, static kServerIdFieldNumber = 1, static kTermFieldNumber = 2, static kLastLogTermFieldNumber = 3, static kLastLogIndexFieldNumber = 4, _unknown_fields_ = {fields_ = 0x0}, server_id_ = 2, term_ = 2116, last_log_term_ = 2114, last_log_index_ = 16690, _cached_size_ = 0, _has_bits_ = {15}, static default_instance_ = 0x1aa8e70}
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x57c121)>
        epoch = 196241
        status = LogCabin::Server::RaftConsensusInternal::Peer::OK
        response = {<google::protobuf::Message> = {<No data fields>}, static kTermFieldNumber = 1, static kGrantedFieldNumber = 2, static kLogOkFieldNumber = 3, _unknown_fields_ = {fields_ = 0x0}, term_ = 0, granted_ = false, log_ok_ = false, _cached_size_ = 0, _has_bits_ = {0}, static default_instance_ = 0x1aaedd0}
        start = {__d = {__r = 31756097569050}}
#8  0x0000000000441898 in LogCabin::Server::RaftConsensus::peerThreadMain (this=0x1aa8190, peer=std::shared_ptr (count 5) 0x7ffcc8012750) at build/Server/RaftConsensus.cc:2086
        now = {__d = {__r = 31756097559680}}
        waitUntil = {__d = {__r = -9223372036854775808}}
        lockGuard = {_M_device = 0x1aa8248, _M_owns = false}
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x57c7ad)>
#9  0x000000000045a309 in std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)(std::shared_ptr<LogCabin::Server::RaftConsensusInternal::Peer>)>::operator() (this=0x7ffcc8006758, __object=0x1aa8190, __args#0=std::shared_ptr (count 5) 0x7ffcc8012750) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
No locals.
#10 0x000000000045a17f in std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)(std::shared_ptr<LogCabin::Server::RaftConsensusInternal::Peer>)>(LogCabin::Server::RaftConsensus*, std::shared_ptr<LogCabin::Server::RaftConsensusInternal::Peer>)>::__call<, 0, 1>(const std::tuple<> &, std::_Index_tuple<0, 1>) (this=0x7ffcc8006758, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
No locals.
#11 0x000000000045a08b in std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)(std::shared_ptr<LogCabin::Server::RaftConsensusInternal::Peer>)>(LogCabin::Server::RaftConsensus*, std::shared_ptr<LogCabin::Server::RaftConsensusInternal::Peer>)>::operator()<>(void) (this=0x7ffcc8006758) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
No locals.
#12 0x0000000000459f08 in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)(std::shared_ptr<LogCabin::Server::RaftConsensusInternal::Peer>)>(LogCabin::Server::RaftConsensus*, std::shared_ptr<LogCabin::Server::RaftConsensusInternal::Peer>)> >::_M_run(void) (this=0x7ffcc8006740) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
No locals.
#13 0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#14 0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#15 0x00007ffd0290e8fd in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 18 (Thread 0x7ffcf97fb700 (LWP 12966)):
#0  0x00007ffd03fa398e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000560e86 in LogCabin::Core::ConditionVariable::wait_until (this=0x7fff826cf6c0, lockGuard=..., abs_time=...) at build/Core/ConditionVariable.cc:127
        wake = {__d = {__r = 31815307524297}}
        mutex = 0x7fff826cf678
        now = {__d = {__r = 31755307528932}}
        wakespec = {tv_sec = 31815, tv_nsec = 307524297}
        r = 32767
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x591bcc)>
#2  0x000000000044ce77 in LogCabin::Core::ConditionVariable::wait_until<LogCabin::Core::Time::CSteadyClock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fff826cf6c0, lockGuard=..., abs_time=...) at ./Core/ConditionVariable.h:105
        __PRETTY_FUNCTION__ = <error reading variable __PRETTY_FUNCTION__ (Cannot access memory at address 0x57c6a0)>
        mutex = @0x7fff826cf678
        stdLockGuard = {_M_device = 0x7fff826cf678, _M_owns = true}
#3  0x000000000045e63a in LogCabin::Server::ServerStats::statsDumperMain (this=0x7fff826cf670) at build/Server/ServerStats.cc:196
        nextDump = {__d = {__r = 31815307524297}}
        lockGuard = {_M_device = 0x0, _M_owns = false}
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x57eb30)>
#4  0x00000000004600c3 in std::_Mem_fn<void (LogCabin::Server::ServerStats::*)()>::operator() (this=0x1aaeb38, __object=0x7fff826cf670) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
No locals.
#5  0x0000000000460038 in std::_Bind<std::_Mem_fn<void (LogCabin::Server::ServerStats::*)()>(LogCabin::Server::ServerStats*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aaeb38, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
No locals.
#6  0x000000000045ffdb in std::_Bind<std::_Mem_fn<void (LogCabin::Server::ServerStats::*)()>(LogCabin::Server::ServerStats*)>::operator()<>(void) (this=0x1aaeb38) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
No locals.
#7  0x000000000045ff1c in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::Server::ServerStats::*)()>(LogCabin::Server::ServerStats*)> >::_M_run(void) (this=0x1aaeb20) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
No locals.
#8  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#9  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#10 0x00007ffd0290e8fd in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 17 (Thread 0x7ffcbbfff700 (LWP 18089)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1aae6e8, lockGuard=...) at build/Core/ConditionVariable.cc:84
        mutex = 0x1aae6a0
        r = 0
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x591c37)>
#2  0x0000000000550b15 in LogCabin::RPC::ThreadDispatchService::workerMain (this=0x1aae680) at build/RPC/ThreadDispatchService.cc:96
        lockGuard = {_M_device = 0x1aae6a0, _M_owns = true}
        rpc = {opaqueRPC = {request = {data = 0x0, length = 0, deleter = 0}, response = {data = 0x0, length = 0, deleter = 0}, socket = std::weak_ptr (empty) 0x0, messageId = 18446744073709551615, responseTarget = 0x0}, active = false, service = 0, serviceSpecificErrorVersion = 0 '\000', opCode = 0}
#3  0x0000000000554a05 in std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>::operator() (this=0x1aaf668, __object=0x1aae680) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
No locals.
#4  0x000000000055497a in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aaf668, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
No locals.
#5  0x000000000055491d in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::operator()<>(void) (this=0x1aaf668) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
No locals.
#6  0x000000000055485e in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)> >::_M_run(void) (this=0x1aaf650) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
No locals.
#7  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#8  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#9  0x00007ffd0290e8fd in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 16 (Thread 0x7ffcf35fe700 (LWP 12968)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1aae6e8, lockGuard=...) at build/Core/ConditionVariable.cc:84
        mutex = 0x1aae6a0
        r = 0
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x591c37)>
#2  0x0000000000550b15 in LogCabin::RPC::ThreadDispatchService::workerMain (this=0x1aae680) at build/RPC/ThreadDispatchService.cc:96
        lockGuard = {_M_device = 0x1aae6a0, _M_owns = true}
        rpc = {opaqueRPC = {request = {data = 0x0, length = 0, deleter = 0}, response = {data = 0x0, length = 0, deleter = 0}, socket = std::weak_ptr (empty) 0x0, messageId = 18446744073709551615, responseTarget = 0x0}, active = false, service = 0, serviceSpecificErrorVersion = 0 '\000', opCode = 0}
#3  0x0000000000554a05 in std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>::operator() (this=0x1aa6218, __object=0x1aae680) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
No locals.
#4  0x000000000055497a in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aa6218, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
No locals.
#5  0x000000000055491d in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::operator()<>(void) (this=0x1aa6218) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
No locals.
#6  0x000000000055485e in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)> >::_M_run(void) (this=0x1aa6200) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
No locals.
#7  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#8  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#9  0x00007ffd0290e8fd in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 15 (Thread 0x7ffcf2bfd700 (LWP 12969)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1aae6e8, lockGuard=...) at build/Core/ConditionVariable.cc:84
        mutex = 0x1aae6a0
        r = 0
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x591c37)>
#2  0x0000000000550b15 in LogCabin::RPC::ThreadDispatchService::workerMain (this=0x1aae680) at build/RPC/ThreadDispatchService.cc:96
        lockGuard = {_M_device = 0x1aae6a0, _M_owns = true}
        rpc = {opaqueRPC = {request = {data = 0x0, length = 0, deleter = 0}, response = {data = 0x0, length = 0, deleter = 0}, socket = std::weak_ptr (empty) 0x0, messageId = 18446744073709551615, responseTarget = 0x0}, active = false, service = 0, serviceSpecificErrorVersion = 0 '\000', opCode = 0}
#3  0x0000000000554a05 in std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>::operator() (this=0x1aa6158, __object=0x1aae680) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
No locals.
#4  0x000000000055497a in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aa6158, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
No locals.
#5  0x000000000055491d in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::operator()<>(void) (this=0x1aa6158) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
No locals.
#6  0x000000000055485e in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)> >::_M_run(void) (this=0x1aa6140) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
No locals.
#7  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#8  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#9  0x00007ffd0290e8fd in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 14 (Thread 0x7ffcf21fc700 (LWP 14481)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1ab0ab8, lockGuard=...) at build/Core/ConditionVariable.cc:84
        mutex = 0x1ab0a70
        r = 0
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x591c37)>
#2  0x0000000000550b15 in LogCabin::RPC::ThreadDispatchService::workerMain (this=0x1ab0a50) at build/RPC/ThreadDispatchService.cc:96
        lockGuard = {_M_device = 0x1ab0a70, _M_owns = true}
        rpc = {opaqueRPC = {request = {data = 0x0, length = 0, deleter = 0}, response = {data = 0x0, length = 0, deleter = 0}, socket = std::weak_ptr (empty) 0x0, messageId = 18446744073709551615, responseTarget = 0x0}, active = false, service = 0, serviceSpecificErrorVersion = 0 '\000', opCode = 0}
#3  0x0000000000554a05 in std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>::operator() (this=0x1ab40b8, __object=0x1ab0a50) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
No locals.
#4  0x000000000055497a in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1ab40b8, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
No locals.
#5  0x000000000055491d in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::operator()<>(void) (this=0x1ab40b8) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
No locals.
#6  0x000000000055485e in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)> >::_M_run(void) (this=0x1ab40a0) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
No locals.
#7  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#8  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#9  0x00007ffd0290e8fd in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 13 (Thread 0x7ffcb97fb700 (LWP 25063)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1aae6e8, lockGuard=...) at build/Core/ConditionVariable.cc:84
        mutex = 0x1aae6a0
        r = 0
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x591c37)>
#2  0x0000000000550b15 in LogCabin::RPC::ThreadDispatchService::workerMain (this=0x1aae680) at build/RPC/ThreadDispatchService.cc:96
        lockGuard = {_M_device = 0x1aae6a0, _M_owns = true}
        rpc = {opaqueRPC = {request = {data = 0x0, length = 0, deleter = 0}, response = {data = 0x0, length = 0, deleter = 0}, socket = std::weak_ptr (empty) 0x0, messageId = 18446744073709551615, responseTarget = 0x0}, active = false, service = 0, serviceSpecificErrorVersion = 0 '\000', opCode = 0}
#3  0x0000000000554a05 in std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>::operator() (this=0x1aaecf8, __object=0x1aae680) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
No locals.
#4  0x000000000055497a in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aaecf8, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
No locals.
#5  0x000000000055491d in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::operator()<>(void) (this=0x1aaecf8) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
No locals.
#6  0x000000000055485e in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)> >::_M_run(void) (this=0x1aaece0) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
No locals.
#7  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#8  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#9  0x00007ffd0290e8fd in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 12 (Thread 0x7ffd0260f700 (LWP 12958)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1aaef40, lockGuard=...) at build/Core/ConditionVariable.cc:84
        mutex = 0x1aaeef8
        r = 0
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x591c37)>
#2  0x0000000000560c57 in LogCabin::Core::ConditionVariable::wait (this=0x1aaef40, lockGuard=...) at build/Core/ConditionVariable.cc:100
        mutex = @0x1aaeef8
        __PRETTY_FUNCTION__ = <error reading variable __PRETTY_FUNCTION__ (Cannot access memory at address 0x591be0)>
        stdLockGuard = {_M_device = 0x1aaeef8, _M_owns = true}
#3  0x00000000004802db in LogCabin::Storage::SegmentedLog::PreparedSegments::waitForDemand (this=0x1aaeef0) at build/Storage/SegmentedLog.cc:138
        lockGuard = {_M_device = 0x0, _M_owns = false}
#4  0x0000000000486e42 in LogCabin::Storage::SegmentedLog::segmentPreparerMain (this=0x1aaee00) at build/Storage/SegmentedLog.cc:1350
        fileId = 0
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x5831d0)>
#5  0x0000000000495231 in std::_Mem_fn<void (LogCabin::Storage::SegmentedLog::*)()>::operator() (this=0x1aaaf78, __object=0x1aaee00) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
No locals.
#6  0x00000000004951a6 in std::_Bind<std::_Mem_fn<void (LogCabin::Storage::SegmentedLog::*)()>(LogCabin::Storage::SegmentedLog*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aaaf78, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
No locals.
#7  0x0000000000495149 in std::_Bind<std::_Mem_fn<void (LogCabin::Storage::SegmentedLog::*)()>(LogCabin::Storage::SegmentedLog*)>::operator()<>(void) (this=0x1aaaf78) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
No locals.
#8  0x000000000049508a in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::Storage::SegmentedLog::*)()>(LogCabin::Storage::SegmentedLog*)> >::_M_run(void) (this=0x1aaaf60) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
No locals.
#9  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#10 0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#11 0x00007ffd0290e8fd in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 11 (Thread 0x7ffcb8dfa700 (LWP 17183)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1ab0ab8, lockGuard=...) at build/Core/ConditionVariable.cc:84
        mutex = 0x1ab0a70
        r = 0
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x591c37)>
#2  0x0000000000550b15 in LogCabin::RPC::ThreadDispatchService::workerMain (this=0x1ab0a50) at build/RPC/ThreadDispatchService.cc:96
        lockGuard = {_M_device = 0x1ab0a70, _M_owns = true}
        rpc = {opaqueRPC = {request = {data = 0x0, length = 0, deleter = 0}, response = {data = 0x0, length = 0, deleter = 0}, socket = std::weak_ptr (empty) 0x0, messageId = 18446744073709551615, responseTarget = 0x0}, active = false, service = 0, serviceSpecificErrorVersion = 0 '\000', opCode = 0}
#3  0x0000000000554a05 in std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>::operator() (this=0x1acfd78, __object=0x1ab0a50) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
No locals.
#4  0x000000000055497a in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1acfd78, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
No locals.
#5  0x000000000055491d in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::operator()<>(void) (this=0x1acfd78) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
No locals.
#6  0x000000000055485e in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)> >::_M_run(void) (this=0x1acfd60) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
No locals.
#7  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#8  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#9  0x00007ffd0290e8fd in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 10 (Thread 0x7ffcbb5fe700 (LWP 18090)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1aae6e8, lockGuard=...) at build/Core/ConditionVariable.cc:84
        mutex = 0x1aae6a0
        r = 0
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x591c37)>
#2  0x0000000000550b15 in LogCabin::RPC::ThreadDispatchService::workerMain (this=0x1aae680) at build/RPC/ThreadDispatchService.cc:96
        lockGuard = {_M_device = 0x1aae6a0, _M_owns = true}
        rpc = {opaqueRPC = {request = {data = 0x0, length = 0, deleter = 0}, response = {data = 0x0, length = 0, deleter = 0}, socket = std::weak_ptr (empty) 0x0, messageId = 18446744073709551615, responseTarget = 0x0}, active = false, service = 0, serviceSpecificErrorVersion = 0 '\000', opCode = 0}
#3  0x0000000000554a05 in std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>::operator() (this=0x1aa5a68, __object=0x1aae680) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
No locals.
#4  0x000000000055497a in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aa5a68, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
No locals.
#5  0x000000000055491d in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::operator()<>(void) (this=0x1aa5a68) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
No locals.
#6  0x000000000055485e in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)> >::_M_run(void) (this=0x1aa5a50) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
No locals.
#7  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#8  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#9  0x00007ffd0290e8fd in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 9 (Thread 0x7ffcbabfd700 (LWP 19137)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1aae6e8, lockGuard=...) at build/Core/ConditionVariable.cc:84
        mutex = 0x1aae6a0
        r = 0
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x591c37)>
#2  0x0000000000550b15 in LogCabin::RPC::ThreadDispatchService::workerMain (this=0x1aae680) at build/RPC/ThreadDispatchService.cc:96
        lockGuard = {_M_device = 0x1aae6a0, _M_owns = true}
        rpc = {opaqueRPC = {request = {data = 0x0, length = 0, deleter = 0}, response = {data = 0x0, length = 0, deleter = 0}, socket = std::weak_ptr (empty) 0x0, messageId = 18446744073709551615, responseTarget = 0x0}, active = false, service = 0, serviceSpecificErrorVersion = 0 '\000', opCode = 0}
#3  0x0000000000554a05 in std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>::operator() (this=0x1aaccb8, __object=0x1aae680) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
No locals.
#4  0x000000000055497a in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aaccb8, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
No locals.
#5  0x000000000055491d in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::operator()<>(void) (this=0x1aaccb8) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
No locals.
#6  0x000000000055485e in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)> >::_M_run(void) (this=0x1aacca0) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
No locals.
#7  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#8  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#9  0x00007ffd0290e8fd in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 8 (Thread 0x7ffca3fff700 (LWP 26705)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1aae6e8, lockGuard=...) at build/Core/ConditionVariable.cc:84
        mutex = 0x1aae6a0
        r = 0
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x591c37)>
#2  0x0000000000550b15 in LogCabin::RPC::ThreadDispatchService::workerMain (this=0x1aae680) at build/RPC/ThreadDispatchService.cc:96
        lockGuard = {_M_device = 0x1aae6a0, _M_owns = true}
        rpc = {opaqueRPC = {request = {data = 0x0, length = 0, deleter = 0}, response = {data = 0x0, length = 0, deleter = 0}, socket = std::weak_ptr (empty) 0x0, messageId = 18446744073709551615, responseTarget = 0x0}, active = false, service = 0, serviceSpecificErrorVersion = 0 '\000', opCode = 0}
#3  0x0000000000554a05 in std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>::operator() (this=0x1af4848, __object=0x1aae680) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
No locals.
#4  0x000000000055497a in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1af4848, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
No locals.
#5  0x000000000055491d in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::operator()<>(void) (this=0x1af4848) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
No locals.
#6  0x000000000055485e in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)> >::_M_run(void) (this=0x1af4830) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
No locals.
#7  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#8  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#9  0x00007ffd0290e8fd in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 7 (Thread 0x7ffcfb5fe700 (LWP 12962)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1aa8290, lockGuard=...) at build/Core/ConditionVariable.cc:84
        mutex = 0x1aa8248
        r = 0
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x591c37)>
#2  0x0000000000560c57 in LogCabin::Core::ConditionVariable::wait (this=0x1aa8290, lockGuard=...) at build/Core/ConditionVariable.cc:100
        mutex = @0x1aa8248
        __PRETTY_FUNCTION__ = <error reading variable __PRETTY_FUNCTION__ (Cannot access memory at address 0x591be0)>
        stdLockGuard = {_M_device = 0x1aa8248, _M_owns = true}
#3  0x0000000000441c28 in LogCabin::Server::RaftConsensus::stepDownThreadMain (this=0x1aa8190) at build/Server/RaftConsensus.cc:2135
        stepDownAt = {__d = {__r = 31743362667143}}
        term = 2114
        epoch = 196206
        lockGuard = {_M_device = 0x0, _M_owns = false}
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x57c680)>
#4  0x000000000045a23f in std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>::operator() (this=0x1aa7188, __object=0x1aa8190) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
No locals.
#5  0x000000000045a0f6 in std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>(LogCabin::Server::RaftConsensus*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aa7188, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
No locals.
#6  0x000000000045a059 in std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>(LogCabin::Server::RaftConsensus*)>::operator()<>(void) (this=0x1aa7188) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
No locals.
#7  0x0000000000459df6 in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>(LogCabin::Server::RaftConsensus*)> >::_M_run(void) (this=0x1aa7170) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
No locals.
#8  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#9  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#10 0x00007ffd0290e8fd in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 6 (Thread 0x7ffcba1fc700 (LWP 19138)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1aae6e8, lockGuard=...) at build/Core/ConditionVariable.cc:84
        mutex = 0x1aae6a0
        r = 0
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x591c37)>
#2  0x0000000000550b15 in LogCabin::RPC::ThreadDispatchService::workerMain (this=0x1aae680) at build/RPC/ThreadDispatchService.cc:96
        lockGuard = {_M_device = 0x1aae6a0, _M_owns = true}
        rpc = {opaqueRPC = {request = {data = 0x0, length = 0, deleter = 0}, response = {data = 0x0, length = 0, deleter = 0}, socket = std::weak_ptr (empty) 0x0, messageId = 18446744073709551615, responseTarget = 0x0}, active = false, service = 0, serviceSpecificErrorVersion = 0 '\000', opCode = 0}
#3  0x0000000000554a05 in std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>::operator() (this=0x1aacbf8, __object=0x1aae680) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
No locals.
#4  0x000000000055497a in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aacbf8, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
No locals.
#5  0x000000000055491d in std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)>::operator()<>(void) (this=0x1aacbf8) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
No locals.
#6  0x000000000055485e in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::RPC::ThreadDispatchService::*)()>(LogCabin::RPC::ThreadDispatchService*)> >::_M_run(void) (this=0x1aacbe0) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
No locals.
#7  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#8  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#9  0x00007ffd0290e8fd in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 5 (Thread 0x7ffcf0dfa700 (LWP 14483)):
#0  0x00007ffd0290eef3 in epoll_wait () from /lib64/libc.so.6
No symbol table info available.
#1  0x00000000005556f0 in LogCabin::Event::Loop::runForever (this=0x7ffcf0df9350) at build/Event/Loop.cc:154
        extraLockGuard = {_M_device = 0x7ffcf0df9470, _M_owns = true}
        events = {{events = 109512359, data = {ptr = 0xf0df957000001ce2, fd = 7394, u32 = 7394, u64 = 17356755797178064098}}}
        r = 31757
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x58f294)>
#2  0x0000000000540e1c in LogCabin::RPC::ClientSession::ClientSession (this=0x7ffcc0000b30, eventLoop=..., address=..., maxMessageLength=1049600, timeout=..., config=...) at build/RPC/ClientSession.cc:301
        loop = {epollfd = 1020, breakTimer = {<LogCabin::Event::Timer> = {<LogCabin::Event::File> = {_vptr.File = 0x8272d0, fd = 1021, ownership = LogCabin::Event::File::CLOSE_ON_DESTROY}, <No data fields>}, <No data fields>}, shouldExit = false, mutex = {_M_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, runningThread = 16, numLocks = 0, numActiveLocks = 0, lockOwner = 0, safeToLock = {cv = {__data = {__lock = 0, __futex = 0, __total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0, __nwaiters = 1, __broadcast_seq = 0}, __size = '\000' <repeats 40 times>, "\001\000\000\000\000\000\000", __align = 0}, callback = {<std::_Maybe_unary_or_binary_function<void>> = {<No data fields>}, <std::_Function_base> = {static _M_max_size = 16, static _M_max_align = 8, _M_functor = {_M_unused = {_M_object = 0x7ffc00000005, _M_const_object = 0x7ffc00000005, _M_function_pointer = 0x7ffc00000005, _M_member_pointer = &virtual table offset 140720308486148, this adjustment 1}, _M_pod_data = "\005\000\000\000\374\177\000\000\001\000\000\000\000\000\000"}, _M_manager = 0}, _M_invoker = 0x70}, notificationCount = {<std::__atomic2::__atomic_base<unsigned long>> = {_M_i = 0}, <No data fields>}, lastWaitUntil = {__d = {__r = 0}}}, unlocked = {cv = {__data = {__lock = 0, __futex = 0, __total_seq = 0, __wakeup_seq = 0, __woken_seq = 0, __mutex = 0x0, __nwaiters = 1, __broadcast_seq = 0}, __size = '\000' <repeats 40 times>, "\001\000\000\000\000\000\000", __align = 0}, callback = {<std::_Maybe_unary_or_binary_function<void>> = {<No data fields>}, <std::_Function_base> = {static _M_max_size = 16, static _M_max_align = 8, _M_functor = {_M_unused = {_M_object = 0x40, _M_const_object = 0x40, _M_function_pointer = 0x40, _M_member_pointer = (void (std::_Undefined_class::*)(std::_Undefined_class *)) 0x40, this adjustment 140723529711648}, _M_pod_data = "@\000\000\000\000\000\000\000 \000\000\300\374\177\000"}, _M_manager = 0}, _M_invoker = 0x2}, notificationCount = {<std::__atomic2::__atomic_base<unsigned long>> = {_M_i = 0}, <No data fields>}, lastWaitUntil = {__d = {__r = 0}}}, extraMutexToSatisfyRaceDetector = {<No data fields>}, breakTimerMonitor = {<LogCabin::Event::File::Monitor> = {_vptr.Monitor = 0x827450, eventLoop = @0x7ffcf0df9350, mutex = {_M_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, file = 0x7ffcf0df9358}, <No data fields>}}
        fileNotifier = {<LogCabin::Event::File> = {_vptr.File = 0x81ded0, fd = 1019, ownership = LogCabin::Event::File::CALLER_CLOSES_FD}, eventLoop = @0x7ffcf0df9350, count = 0}
        timerNotifier = {<LogCabin::Event::Timer> = {<LogCabin::Event::File> = {_vptr.File = 0x826af0, fd = 1022, ownership = LogCabin::Event::File::CLOSE_ON_DESTROY}, <No data fields>}, eventLoop = @0x7ffcf0df9350}
        fileMonitor = {_vptr.Monitor = 0x827290, eventLoop = @0x7ffcf0df9350, mutex = {_M_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, file = 0x7ffcf0df9540}
        timerMonitor = {<LogCabin::Event::File::Monitor> = {_vptr.Monitor = 0x827450, eventLoop = @0x7ffcf0df9350, mutex = {_M_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, file = 0x7ffcf0df9570}, <No data fields>}
        fd = 1019
        waiting = true
        r = -1
#3  0x000000000054139c in LogCabin::RPC::ClientSession::makeSession (eventLoop=..., address=..., maxMessageLength=1049600, timeout=..., config=...) at build/RPC/ClientSession.cc:346
        session = std::shared_ptr (empty) 0x0
#4  0x00000000004c2525 in LogCabin::Client::SessionManager::createSession (this=0x1aa8230, address=..., timeout=..., clusterUUID=0x7fff826cfa30, serverId=0x7ffcf0df9a00) at build/Client/SessionManager.cc:47
        session = std::shared_ptr (empty) 0x0
        rpc = {service = 38800, opCode = 61663, opaqueRPC = {mutex = {_M_mutex = {__data = {__lock = 4385355, __count = 0, __owner = -253782064, __nusers = 32764, __kind = 8577888, __spins = 0, __list = {__prev = 0x7ffcf0df97b0, __next = 0x4305ba}}, __size = "K\352B\000\000\000\000\000З\337\360\374\177\000\000`\343\202\000\000\000\000\000\260\227\337\360\374\177\000\000\272\005C\000\000\000\000", __align = 4385355}}, session = std::shared_ptr (count 0) 0x82e4a0, responseToken = 140724349671376, status = 4393669, reply = {data = 0x7ffcf0df9820, length = 140724349671440, deleter = 0x7ffcf0df9850}, errorMessage = Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 558, in to_string
    return self.val['_M_dataplus']['_M_p'].lazy_string (length = len)
RuntimeError: Cannot access memory at address 0x566bcc
}}
        response = {<google::protobuf::Message> = {<No data fields>}, static kClusterUuidFieldNumber = 1, static kServerIdFieldNumber = 2, static kOkFieldNumber = 3, static kErrorFieldNumber = 4, _unknown_fields_ = {fields_ = 0x1aacff8}, cluster_uuid_ = 0x58d0da, server_id_ = 171804183902, error_ = 0x1aac510, ok_ = 208, _cached_size_ = 0, _has_bits_ = {8577888}, static default_instance_ = 0x1ab0b80}
        status = LogCabin::RPC::ClientRPC::OK
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x587c10)>
        request = {<google::protobuf::Message> = {<No data fields>}, static kClusterUuidFieldNumber = 1, static kServerIdFieldNumber = 2, _unknown_fields_ = {fields_ = 0x53c07a}, cluster_uuid_ = 0x7ffcc8004d10, server_id_ = 140724349671760, _cached_size_ = -185515627, _has_bits_ = {7394}, static default_instance_ = 0x1ab1790}
        error = {<google::protobuf::Message> = {<No data fields>}, static UNKNOWN = LogCabin::Protocol::Client::Error_Code_UNKNOWN, static NOT_LEADER = LogCabin::Protocol::Client::Error_Code_NOT_LEADER, static Code_MIN = LogCabin::Protocol::Client::Error_Code_UNKNOWN, static Code_MAX = LogCabin::Protocol::Client::Error_Code_NOT_LEADER, static Code_ARRAYSIZE = 2, static kErrorCodeFieldNumber = 1, static kLeaderHintFieldNumber = 2, _unknown_fields_ = {fields_ = 0x1aacff8}, leader_hint_ = 0x3, error_code_ = 42948152, _cached_size_ = 30, _has_bits_ = {2684759619}, static default_instance_ = 0x1ab1470}
#5  0x0000000000438194 in LogCabin::Server::RaftConsensusInternal::Peer::getSession (this=0x7ffcc8004d00, lockGuard=...) at build/Server/RaftConsensus.cc:339
        timeout = {__d = {__r = 31761097638293}}
        unlockGuard = {guard = @0x7ffcf0df9d30}
        target = {originalString = "192.168.8.35", hosts = std::vector of length 1, capacity 1 = {{first = "192.168.8.35", second = "5254"}}, storage = {ss_family = 2, __ss_align = 0, __ss_padding = '\000' <repeats 111 times>}, len = 16}
        peerId = {mutex = {_M_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, empty = false, value = 3}
#6  0x00000000004379f3 in LogCabin::Server::RaftConsensusInternal::Peer::callRPC (this=0x7ffcc8004d00, opCode=LogCabin::Protocol::Raft::REQUEST_VOTE, request=..., response=..., lockGuard=...) at build/Server/RaftConsensus.cc:275
        unlockGuard = {guard = @0x1ce1caeb4dad}
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x57ce93)>
#7  0x0000000000444f5e in LogCabin::Server::RaftConsensus::requestVote (this=0x1aa8190, lockGuard=..., peer=...) at build/Server/RaftConsensus.cc:2769
        request = {<google::protobuf::Message> = {<No data fields>}, static kServerIdFieldNumber = 1, static kTermFieldNumber = 2, static kLastLogTermFieldNumber = 3, static kLastLogIndexFieldNumber = 4, _unknown_fields_ = {fields_ = 0x0}, server_id_ = 2, term_ = 2116, last_log_term_ = 2114, last_log_index_ = 16690, _cached_size_ = 0, _has_bits_ = {15}, static default_instance_ = 0x1aa8e70}
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x57c121)>
        epoch = 196241
        status = LogCabin::Server::RaftConsensusInternal::Peer::OK
        response = {<google::protobuf::Message> = {<No data fields>}, static kTermFieldNumber = 1, static kGrantedFieldNumber = 2, static kLogOkFieldNumber = 3, _unknown_fields_ = {fields_ = 0x0}, term_ = 0, granted_ = false, log_ok_ = false, _cached_size_ = 0, _has_bits_ = {0}, static default_instance_ = 0x1aaedd0}
        start = {__d = {__r = 31756097637805}}
#8  0x0000000000441898 in LogCabin::Server::RaftConsensus::peerThreadMain (this=0x1aa8190, peer=std::shared_ptr (count 5) 0x7ffcc8004d00) at build/Server/RaftConsensus.cc:2086
        now = {__d = {__r = 31756097633617}}
        waitUntil = {__d = {__r = -9223372036854775808}}
        lockGuard = {_M_device = 0x1aa8248, _M_owns = false}
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x57c7ad)>
#9  0x000000000045a309 in std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)(std::shared_ptr<LogCabin::Server::RaftConsensusInternal::Peer>)>::operator() (this=0x7ffcc8007c88, __object=0x1aa8190, __args#0=std::shared_ptr (count 5) 0x7ffcc8004d00) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
No locals.
#10 0x000000000045a17f in std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)(std::shared_ptr<LogCabin::Server::RaftConsensusInternal::Peer>)>(LogCabin::Server::RaftConsensus*, std::shared_ptr<LogCabin::Server::RaftConsensusInternal::Peer>)>::__call<, 0, 1>(const std::tuple<> &, std::_Index_tuple<0, 1>) (this=0x7ffcc8007c88, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
No locals.
#11 0x000000000045a08b in std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)(std::shared_ptr<LogCabin::Server::RaftConsensusInternal::Peer>)>(LogCabin::Server::RaftConsensus*, std::shared_ptr<LogCabin::Server::RaftConsensusInternal::Peer>)>::operator()<>(void) (this=0x7ffcc8007c88) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
No locals.
#12 0x0000000000459f08 in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)(std::shared_ptr<LogCabin::Server::RaftConsensusInternal::Peer>)>(LogCabin::Server::RaftConsensus*, std::shared_ptr<LogCabin::Server::RaftConsensusInternal::Peer>)> >::_M_run(void) (this=0x7ffcc8007c70) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
No locals.
#13 0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#14 0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#15 0x00007ffd0290e8fd in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 4 (Thread 0x7ffcfa1fc700 (LWP 12965)):
#0  0x00007ffd03fa398e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000560e86 in LogCabin::Core::ConditionVariable::wait_until (this=0x1ab36c0, lockGuard=..., abs_time=...) at build/Core/ConditionVariable.cc:127
        wake = {__d = {__r = 33435465781129}}
        mutex = 0x1ab35b8
        now = {__d = {__r = 29835465781129}}
        wakespec = {tv_sec = 33435, tv_nsec = 465781129}
        r = 0
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x591bcc)>
#2  0x000000000044ce77 in LogCabin::Core::ConditionVariable::wait_until<LogCabin::Core::Time::CSteadyClock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x1ab36c0, lockGuard=..., abs_time=...) at ./Core/ConditionVariable.h:105
        __PRETTY_FUNCTION__ = <error reading variable __PRETTY_FUNCTION__ (Cannot access memory at address 0x57c6a0)>
        mutex = @0x1ab35b8
        stdLockGuard = {_M_device = 0x1ab35b8, _M_owns = true}
#3  0x00000000004640cf in LogCabin::Server::StateMachine::snapshotWatchdogThreadMain (this=0x1ab3570) at build/Server/StateMachine.cc:714
        waitUntil = {__d = {__r = 9223372036854775807}}
        now = {__d = {__r = 29835465780226}}
        startProgress = 0
        startTime = {__d = {__r = -9223372036854775808}}
        zero = {__r = 0}
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x57fcf0)>
        lockGuard = {_M_device = 0x0, _M_owns = false}
        tracking = 18446744073709551615
#4  0x000000000046c015 in std::_Mem_fn<void (LogCabin::Server::StateMachine::*)()>::operator() (this=0x1ab19f8, __object=0x1ab3570) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
No locals.
#5  0x000000000046bf8a in std::_Bind<std::_Mem_fn<void (LogCabin::Server::StateMachine::*)()>(LogCabin::Server::StateMachine*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1ab19f8, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
No locals.
#6  0x000000000046bf2d in std::_Bind<std::_Mem_fn<void (LogCabin::Server::StateMachine::*)()>(LogCabin::Server::StateMachine*)>::operator()<>(void) (this=0x1ab19f8) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
No locals.
#7  0x000000000046be6e in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::Server::StateMachine::*)()>(LogCabin::Server::StateMachine*)> >::_M_run(void) (this=0x1ab19e0) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
No locals.
#8  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#9  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#10 0x00007ffd0290e8fd in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 3 (Thread 0x7ffcfabfd700 (LWP 12963)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1aa8290, lockGuard=...) at build/Core/ConditionVariable.cc:84
        mutex = 0x1aa8248
        r = 0
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x591c37)>
#2  0x0000000000560c57 in LogCabin::Core::ConditionVariable::wait (this=0x1aa8290, lockGuard=...) at build/Core/ConditionVariable.cc:100
        mutex = @0x1aa8248
        __PRETTY_FUNCTION__ = <error reading variable __PRETTY_FUNCTION__ (Cannot access memory at address 0x591be0)>
        stdLockGuard = {_M_device = 0x1aa8248, _M_owns = true}
#3  0x000000000043d450 in LogCabin::Server::RaftConsensus::getNextEntry (this=0x1aa8190, lastIndex=16689) at build/Server/RaftConsensus.cc:1242
        lockGuard = {_M_device = 0x0, _M_owns = false}
        nextIndex = 16690
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x57cc64)>
#4  0x00000000004626e9 in LogCabin::Server::StateMachine::applyThreadMain (this=0x1ab3570) at build/Server/StateMachine.cc:391
        entry = {index = 16689, type = LogCabin::Server::RaftConsensus::Entry::DATA, command = {data = 0x7ffcec001970, length = 147, deleter = 0x44c877 <LogCabin::Core::Buffer::deleteArrayFn<char>(void*)>}, snapshotReader = std::tuple containing = {[1] = , [2] = {<std::default_delete<LogCabin::Storage::SnapshotFile::Reader>> = {<No data fields>}, <No data fields>}}, clusterTime = 15598993486651}
        lockGuard = {_M_device = @0x1ab35b8}
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x57ffe0)>
#5  0x000000000046c015 in std::_Mem_fn<void (LogCabin::Server::StateMachine::*)()>::operator() (this=0x1aa77f8, __object=0x1ab3570) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
No locals.
#6  0x000000000046bf8a in std::_Bind<std::_Mem_fn<void (LogCabin::Server::StateMachine::*)()>(LogCabin::Server::StateMachine*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aa77f8, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
No locals.
#7  0x000000000046bf2d in std::_Bind<std::_Mem_fn<void (LogCabin::Server::StateMachine::*)()>(LogCabin::Server::StateMachine*)>::operator()<>(void) (this=0x1aa77f8) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
No locals.
#8  0x000000000046be6e in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::Server::StateMachine::*)()>(LogCabin::Server::StateMachine*)> >::_M_run(void) (this=0x1aa77e0) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
No locals.
#9  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#10 0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#11 0x00007ffd0290e8fd in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 2 (Thread 0x7ffcfbfff700 (LWP 12961)):
#0  0x00007ffd03fa35bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
No symbol table info available.
#1  0x0000000000560b49 in LogCabin::Core::ConditionVariable::wait (this=0x1aa8290, lockGuard=...) at build/Core/ConditionVariable.cc:84
        mutex = 0x1aa8248
        r = 0
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x591c37)>
#2  0x0000000000560c57 in LogCabin::Core::ConditionVariable::wait (this=0x1aa8290, lockGuard=...) at build/Core/ConditionVariable.cc:100
        mutex = @0x1aa8248
        __PRETTY_FUNCTION__ = <error reading variable __PRETTY_FUNCTION__ (Cannot access memory at address 0x591be0)>
        stdLockGuard = {_M_device = 0x1aa8248, _M_owns = true}
#3  0x0000000000441214 in LogCabin::Server::RaftConsensus::stateMachineUpdaterThreadMain (this=0x1aa8190) at build/Server/RaftConsensus.cc:2009
        now = {__d = {__r = 31755374422183}}
        lockGuard = {_M_device = 0x0, _M_owns = false}
        lastVersionCommitted = 2
        backoffUntil = {__d = {__r = 1425887726820}}
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x57c830)>
#4  0x000000000045a23f in std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>::operator() (this=0x1aa70b8, __object=0x1aa8190) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:552
No locals.
#5  0x000000000045a0f6 in std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>(LogCabin::Server::RaftConsensus*)>::__call<, 0>(const std::tuple<> &, std::_Index_tuple<0>) (this=0x1aa70b8, __args=Traceback (most recent call last):
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 262, in children
    return self._iterator (self.val)
  File "/usr/lib64/../share/gdb/python/libstdcxx/v6/printers.py", line 220, in __init__
    raise "Top of tuple tree does not consist of a single node."
TypeError: exceptions must be old-style classes or derived from BaseException, not str
std::tuple containing) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1137
No locals.
#6  0x000000000045a059 in std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>(LogCabin::Server::RaftConsensus*)>::operator()<>(void) (this=0x1aa70b8) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/tr1_impl/functional:1191
No locals.
#7  0x0000000000459df6 in _Impl<std::_Bind<std::_Mem_fn<void (LogCabin::Server::RaftConsensus::*)()>(LogCabin::Server::RaftConsensus*)> >::_M_run(void) (this=0x1aa70a0) at /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/thread:114
No locals.
#8  0x00007ffd0310a470 in ?? () from /usr/lib64/libstdc++.so.6
No symbol table info available.
#9  0x00007ffd03f9f9d1 in start_thread () from /lib64/libpthread.so.0
No symbol table info available.
#10 0x00007ffd0290e8fd in clone () from /lib64/libc.so.6
No symbol table info available.

Thread 1 (Thread 0x7ffd043c4720 (LWP 12956)):
#0  0x00007ffd02858625 in raise () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007ffd02859e05 in abort () from /lib64/libc.so.6
No symbol table info available.
#2  0x0000000000544a45 in LogCabin::RPC::(anonymous namespace)::dupOrPanic (oldfd=1023) at build/RPC/MessageSocket.cc:43
        newfd = -1
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x58e200)>
#3  0x00000000005450d5 in LogCabin::RPC::MessageSocket::MessageSocket (this=0x1b15e00, handler=..., eventLoop=..., fd=1023, maxMessageLength=1049600) at build/RPC/MessageSocket.cc:186
No locals.
#4  0x000000000054885b in LogCabin::RPC::OpaqueServer::SocketWithHandler::SocketWithHandler (this=0x1b15de0, server=0x1aab7e8, fd=1023) at build/RPC/OpaqueServer.cc:115
No locals.
#5  0x0000000000548789 in LogCabin::RPC::OpaqueServer::SocketWithHandler::make (server=0x1aab7e8, fd=1023) at build/RPC/OpaqueServer.cc:106
        socket = std::shared_ptr (count -2106788351) 0x7fff826cf5b0
#6  0x0000000000548a05 in LogCabin::RPC::OpaqueServer::BoundListener::handleFileEvent (this=0x1aa8470, events=1) at build/RPC/OpaqueServer.cc:146
        clientfd = 1023
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x58e600)>
#7  0x00000000005557c7 in LogCabin::Event::Loop::runForever (this=0x7fff826cf788) at build/Event/Loop.cc:162
        file = @0x1aa8470
        i = 0
        extraLockGuard = {_M_device = 0x7fff826cf8a8, _M_owns = true}
        events = {{events = 1, data = {ptr = 0x1aa8470, fd = 27952240, u32 = 27952240, u64 = 27952240}}}
        r = 1
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x58f294)>
#8  0x000000000043466a in LogCabin::Server::Globals::run (this=0x7fff826cf630) at build/Server/Globals.cc:197
No locals.
#9  0x000000000042db04 in main (argc=8, argv=0x7fff826d01a8) at build/Server/Main.cc:352
        globals = {config = {delimiter = "=", comment = "#", contents = std::map with 10 elements = {["electionTimeoutMilliseconds"] = "5000", ["heartbeatPeriodMilliseconds"] = "2500", ["listenAddresses"] = "192.168.8.34", ["maxThreads"] = "128", ["rpcFailureBackoffMilliseconds"] = "2500", ["serverId"] = "2", ["storageModule"] = "Segmented", ["storagePath"] = "/var/lib/logcabin", ["tcpConnectTimeoutMilliseconds"] = "1000", ["tcpHeartbeatTimeoutMilliseconds"] = "2500"}}, serverStats = {globals = @0x7fff826cf630, mutex = {m = {_M_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 1, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 12 times>, "\001", '\000' <repeats 26 times>, __align = 0}}, callback = {<std::_Maybe_unary_or_binary_function<void>> = {<No data fields>}, <std::_Function_base> = {static _M_max_size = 16, static _M_max_align = 8, _M_functor = {_M_unused = {_M_object = 0x24, _M_const_object = 0x24, _M_function_pointer = 0x24, _M_member_pointer = (void (std::_Undefined_class::*)(std::_Undefined_class *)) 0x24, this adjustment 36}, _M_pod_data = "$\000\000\000\000\000\000\000$\000\000\000\000\000\000"}, _M_manager = 0}, _M_invoker = 0x7fff826cff10}}, statsDumpRequested = {cv = {__data = {__lock = 0, __futex = 1025, __total_seq = 513, __wakeup_seq = 512, __woken_seq = 512, __mutex = 0x7fff826cf678, __nwaiters = 3, __broadcast_seq = 0}, __size = "\000\000\000\000\001\004\000\000\001\002\000\000\000\000\000\000\000\002\000\000\000\000\000\000\000\002\000\000\000\000\000\000x\366l\202\377\177\000\000\003\000\000\000\000\000\000", __align = 4402341478400}, callback = {<std::_Maybe_unary_or_binary_function<void>> = {<No data fields>}, <std::_Function_base> = {static _M_max_size = 16, static _M_max_align = 8, _M_functor = {_M_unused = {_M_object = 0x0, _M_const_object = 0x0, _M_function_pointer = 0, _M_member_pointer = NULL}, _M_pod_data = "\000\000\000\000\000\000\000\000f2\034\004\375\177\000"}, _M_manager = 0}, _M_invoker = 0x7fff826cff38}, notificationCount = {<std::__atomic2::__atomic_base<unsigned long>> = {_M_i = 0}, <No data fields>}, lastWaitUntil = {__d = {__r = 31815307524297}}}, exiting = false, isStatsDumpRequested = false, lastDumped = {__d = {__r = 31755307524297}}, stats = {<google::protobuf::Message> = {<No data fields>}, static kServerIdFieldNumber = 1, static kAddressesFieldNumber = 2, static kStartAtFieldNumber = 3, static kEndAtFieldNumber = 4, static kRaftFieldNumber = 11, static kStorageFieldNumber = 12, static kStateMachineFieldNumber = 13, _unknown_fields_ = {fields_ = 0x0}, server_id_ = 2, addresses_ = 0x1aa9d80, start_at_ = 0, end_at_ = 0, raft_ = 0x0, storage_ = 0x0, state_machine_ = 0x0, _cached_size_ = 0, _has_bits_ = {3}, static default_instance_ = 0x1aaacb0}, deferred = std::tuple containing = {[1] = , [2] = {<std::default_delete<LogCabin::Server::ServerStats::Deferred>> = {<No data fields>}, <No data fields>}}}, eventLoop = {epollfd = 4, breakTimer = {<LogCabin::Event::Timer> = {<LogCabin::Event::File> = {_vptr.File = 0x8272d0, fd = 5, ownership = LogCabin::Event::File::CLOSE_ON_DESTROY}, <No data fields>}, <No data fields>}, shouldExit = false, mutex = {_M_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, runningThread = 1, numLocks = 0, numActiveLocks = 0, lockOwner = 0, safeToLock = {cv = {__data = {__lock = 0, __futex = 18482, __total_seq = 9241, __wakeup_seq = 9241, __woken_seq = 9241, __mutex = 0x7fff826cf7a8, __nwaiters = 1, __broadcast_seq = 0}, __size = "\000\000\000\000\062H\000\000\031$\000\000\000\000\000\000\031$\000\000\000\000\000\000\031$\000\000\000\000\000\000\250\367l\202\377\177\000\000\001\000\000\000\000\000\000", __align = 79379585564672}, callback = {<std::_Maybe_unary_or_binary_function<void>> = {<No data fields>}, <std::_Function_base> = {static _M_max_size = 16, static _M_max_align = 8, _M_functor = {_M_unused = {_M_object = 0x7ffd043c99b0, _M_const_object = 0x7ffd043c99b0, _M_function_pointer = 0x7ffd043c99b0, _M_member_pointer = (void (std::_Undefined_class::*)(std::_Undefined_class *)) 0x7ffd043c99b0, this adjustment 140724654300714}, _M_pod_data = "\260\231<\004\375\177\000\000*\336\a\003\375\177\000"}, _M_manager = 0}, _M_invoker = 0x7ffd0305f308}, notificationCount = {<std::__atomic2::__atomic_base<unsigned long>> = {_M_i = 9854}, <No data fields>}, lastWaitUntil = {__d = {__r = 0}}}, unlocked = {cv = {__data = {__lock = 0, __futex = 13598, __total_seq = 6799, __wakeup_seq = 6799, __woken_seq = 6799, __mutex = 0x7fff826cf7a8, __nwaiters = 1, __broadcast_seq = 0}, __size = "\000\000\000\000\036\065\000\000\217\032\000\000\000\000\000\000\217\032\000\000\000\000\000\000\217\032\000\000\000\000\000\000\250\367l\202\377\177\000\000\001\000\000\000\000\000\000", __align = 58402965291008}, callback = {<std::_Maybe_unary_or_binary_function<void>> = {<No data fields>}, <std::_Function_base> = {static _M_max_size = 16, static _M_max_align = 8, _M_functor = {_M_unused = {_M_object = 0x7ffd043c6030, _M_const_object = 0x7ffd043c6030, _M_function_pointer = 0x7ffd043c6030, _M_member_pointer = (void (std::_Undefined_class::*)(std::_Undefined_class *)) 0x7ffd043c6030, this adjustment 3039603026}, _M_pod_data = "0`<\004\375\177\000\000R\251,\265\000\000\000"}, _M_manager = 0}, _M_invoker = 0}, notificationCount = {<std::__atomic2::__atomic_base<unsigned long>> = {_M_i = 48826}, <No data fields>}, lastWaitUntil = {__d = {__r = 0}}}, extraMutexToSatisfyRaceDetector = {<No data fields>}, breakTimerMonitor = {<LogCabin::Event::File::Monitor> = {_vptr.Monitor = 0x827450, eventLoop = @0x7fff826cf788, mutex = {_M_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, file = 0x7fff826cf790}, <No data fields>}}, sigIntBlocker = {signalNumber = 2, isBlocked = true, shouldLeaveBlocked = true}, sigTermBlocker = {signalNumber = 15, isBlocked = true, shouldLeaveBlocked = true}, sigUsr1Blocker = {signalNumber = 10, isBlocked = true, shouldLeaveBlocked = true}, sigUsr2Blocker = {signalNumber = 12, isBlocked = true, shouldLeaveBlocked = true}, sigIntHandler = {<LogCabin::Event::Signal> = {<LogCabin::Event::File> = {_vptr.File = 0x81e370, fd = 6, ownership = LogCabin::Event::File::CLOSE_ON_DESTROY}, signalNumber = 2}, eventLoop = @0x7fff826cf788}, sigIntMonitor = {<LogCabin::Event::File::Monitor> = {_vptr.Monitor = 0x827370, eventLoop = @0x7fff826cf788, mutex = {_M_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, file = 0x7fff826cf910}, <No data fields>}, sigTermHandler = {<LogCabin::Event::Signal> = {<LogCabin::Event::File> = {_vptr.File = 0x81e370, fd = 7, ownership = LogCabin::Event::File::CLOSE_ON_DESTROY}, signalNumber = 15}, eventLoop = @0x7fff826cf788}, sigTermMonitor = {<LogCabin::Event::File::Monitor> = {_vptr.Monitor = 0x827370, eventLoop = @0x7fff826cf788, mutex = {_M_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, file = 0x7fff826cf970}, <No data fields>}, sigUsr2Handler = {<LogCabin::Event::Signal> = {<LogCabin::Event::File> = {_vptr.File = 0x81e330, fd = 8, ownership = LogCabin::Event::File::CLOSE_ON_DESTROY}, signalNumber = 12}, eventLoop = @0x7fff826cf788}, sigUsr2Monitor = {<LogCabin::Event::File::Monitor> = {_vptr.Monitor = 0x827370, eventLoop = @0x7fff826cf788, mutex = {_M_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, file = 0x7fff826cf9d0}, <No data fields>}, clusterUUID = {mutex = {_M_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}, empty = true, value = ""}, serverId = 2, raft = std::shared_ptr (count 2) 0x1aa8190, stateMachine = std::shared_ptr (count 1) 0x1ab3570, controlService = std::shared_ptr (count 2) 0x1ab0d70, raftService = std::shared_ptr (count 2) 0x1ab0ef0, clientService = std::shared_ptr (count 2) 0x1aad4a0, rpcServer = std::tuple containing = {[1] = , [2] = {<std::default_delete<LogCabin::RPC::Server>> = {<No data fields>}, <No data fields>}}}
        options = {argc = @0x7fff826cf62c, argv = @0x7fff826cf620, bootstrap = false, configFilename = "/etc/logcabin.conf", daemon = true, debugLogFilename = "/var/log/logcabin/logcabin.log", pidFilename = "/var/run/logcabin.pid", testConfig = false}
        pidFile = {filename = "/var/run/logcabin.pid", written = 12956}
        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot access memory at address 0x578964)>


portion of the log pre-crash:

2015-11-03 04:01:54.888579 Server/RaftConsensus.cc:2862 in startNewElection() NOTICE[2:startNewElection]: Running for election in term 2106 (haven't heard from leader 1 lately)
2015-11-03 04:01:54.888608 Server/RaftConsensus.cc:2846 in printElectionState() NOTICE[2:startNewElection]: server=2, term=2106, state=CANDIDATE, leader=0, vote=2
2015-11-03 04:01:54.888635 Storage/SegmentedLog.cc:778 in updateMetadata() NOTICE[2:startNewElection]: Writing new storage metadata (version 2212) to metadata2
2015-11-03 04:01:59.892339 Server/RaftConsensus.cc:294 in callRPC() WARNING[2:Peer(1)]: RPC to server failed: Server 192.168.8.33 (resolved to 192.168.8.33:5254) timed out
2015-11-03 04:02:04.316247 Server/RaftConsensus.cc:2867 in startNewElection() NOTICE[2:startNewElection]: Running for election in term 2107 (previous candidacy for term 2106 timed out)
2015-11-03 04:02:04.316283 Server/RaftConsensus.cc:2846 in printElectionState() NOTICE[2:startNewElection]: server=2, term=2107, state=CANDIDATE, leader=0, vote=2
2015-11-03 04:02:04.316302 Storage/SegmentedLog.cc:778 in updateMetadata() NOTICE[2:startNewElection]: Writing new storage metadata (version 2213) to metadata1
2015-11-03 04:02:10.088625 Server/RaftConsensus.cc:2867 in startNewElection() NOTICE[2:startNewElection]: Running for election in term 2108 (previous candidacy for term 2107 timed out)
2015-11-03 04:02:10.088664 Server/RaftConsensus.cc:2846 in printElectionState() NOTICE[2:startNewElection]: server=2, term=2108, state=CANDIDATE, leader=0, vote=2
2015-11-03 04:02:10.088684 Storage/SegmentedLog.cc:778 in updateMetadata() NOTICE[2:startNewElection]: Writing new storage metadata (version 2214) to metadata2
2015-11-03 04:02:16.989516 Server/RaftConsensus.cc:2867 in startNewElection() NOTICE[2:startNewElection]: Running for election in term 2109 (previous candidacy for term 2108 timed out)
2015-11-03 04:02:16.989555 Server/RaftConsensus.cc:2846 in printElectionState() NOTICE[2:startNewElection]: server=2, term=2109, state=CANDIDATE, leader=0, vote=2
2015-11-03 04:02:16.989574 Storage/SegmentedLog.cc:778 in updateMetadata() NOTICE[2:startNewElection]: Writing new storage metadata (version 2215) to metadata1
2015-11-03 04:02:23.251729 Server/RaftConsensus.cc:2867 in startNewElection() NOTICE[2:startNewElection]: Running for election in term 2110 (previous candidacy for term 2109 timed out)
2015-11-03 04:02:23.251767 Server/RaftConsensus.cc:2846 in printElectionState() NOTICE[2:startNewElection]: server=2, term=2110, state=CANDIDATE, leader=0, vote=2
2015-11-03 04:02:23.251786 Storage/SegmentedLog.cc:778 in updateMetadata() NOTICE[2:startNewElection]: Writing new storage metadata (version 2216) to metadata2
2015-11-03 04:02:32.188413 Server/RaftConsensus.cc:2867 in startNewElection() NOTICE[2:startNewElection]: Running for election in term 2111 (previous candidacy for term 2110 timed out)
2015-11-03 04:02:32.188450 Server/RaftConsensus.cc:2846 in printElectionState() NOTICE[2:startNewElection]: server=2, term=2111, state=CANDIDATE, leader=0, vote=2
2015-11-03 04:02:32.188470 Storage/SegmentedLog.cc:778 in updateMetadata() NOTICE[2:startNewElection]: Writing new storage metadata (version 2217) to metadata1
2015-11-03 04:02:39.378282 Server/ServerStats.cc:148 in dumpToDebugLog() NOTICE[2:StatsDumper]: ServerStats:
server_id: 2
addresses: "192.168.8.34"
start_at: 1446523359378114569
end_at: 1446523359378156677
raft {
  current_term: 2111
  state: CANDIDATE
  commit_index: 16628
  last_log_index: 16628
  leader_id: 0
  voted_for: 2
  start_election_at: 1446523359635761560
  withhold_votes_until: 1446523314194397912
  cluster_time: 15625371915054
  cluster_time_epoch: 15572181106836
  last_snapshot_index: 0
  last_snapshot_bytes: 0
  log_start_index: 1
  log_bytes: 3452431
  last_snapshot_term: 0
  last_snapshot_cluster_time: 0
  num_entries_truncated: 19
  peer {
    server_id: 1
    addresses: "192.168.8.33"
    old_member: true
    new_member: false
    staging_member: false
    request_vote_done: false
    have_vote: false
    backoff_until: 1446523359894007662
  }
  peer {
    server_id: 2
    addresses: "192.168.8.34"
    old_member: true
    new_member: false
    staging_member: false
  }
  peer {
    server_id: 3
    addresses: "192.168.8.35"
    old_member: true
    new_member: false
    staging_member: false
    request_vote_done: false
    have_vote: false
    backoff_until: 1446523359893975396
  }
}
storage {
  num_segments: 20
  open_segment_bytes: 96090
  metadata_version: 2217
  metadata_write_nanos {
    average: 3694991.9314388814
    count: 2217
    ewma2: 3681576.8659366136
    ewma4: 3580417.2292418182
    exceptional_count: 0
    last: 3490669
    min: 322927
    max: 62131317
    sum: 8191797112
    stddev: 3985141.7942886911
  }
  filesystem_ops_nanos {
    average: 792110.22097167512
    count: 15993
    ewma2: 326951.25122888724
    ewma4: 398450.30434953765
    exceptional_count: 0
    last: 197691
    min: 167693
    max: 158011023
    sum: 12668218764
    stddev: 3149316.1602298478
  }
}
state_machine {
  snapshotting: false
  last_applied: 16628
  num_sessions: 3
  num_snapshots_attempted: 0
  num_snapshots_failed: 0
  num_redundant_advance_version_entries: 2
  num_rejected_advance_version_entries: 0
  num_successful_advance_version_entries: 1
  num_total_advance_version_entries: 3
  min_supported_version: 1
  max_supported_version: 2
  running_version: 2
  tree {
    num_conditions_checked: 15206
    num_conditions_failed: 1213
    num_make_directory_attempted: 21
    num_make_directory_success: 21
    num_list_directory_attempted: 19087
    num_list_directory_success: 19087
    num_remove_directory_attempted: 0
    num_remove_directory_parent_not_found: 0
    num_remove_directory_target_not_found: 0
    num_remove_directory_done: 0
    num_remove_directory_success: 0
    num_write_attempted: 15152
    num_write_success: 15152
    num_read_attempted: 69894
    num_read_success: 69676
    num_remove_file_attempted: 0
    num_remove_file_parent_not_found: 0
    num_remove_file_target_not_found: 0
    num_remove_file_done: 0
    num_remove_file_success: 0
  }
  num_unknown_requests: 0
  may_snapshot_at: -7776880372783699941
}

2015-11-03 04:02:39.635864 Server/RaftConsensus.cc:2867 in startNewElection() NOTICE[2:startNewElection]: Running for election in term 2112 (previous candidacy for term 2111 timed out)
2015-11-03 04:02:39.635885 Server/RaftConsensus.cc:2846 in printElectionState() NOTICE[2:startNewElection]: server=2, term=2112, state=CANDIDATE, leader=0, vote=2
2015-11-03 04:02:39.635909 Storage/SegmentedLog.cc:778 in updateMetadata() NOTICE[2:startNewElection]: Writing new storage metadata (version 2218) to metadata2
2015-11-03 04:02:49.482011 Server/RaftConsensus.cc:2867 in startNewElection() NOTICE[2:startNewElection]: Running for election in term 2113 (previous candidacy for term 2112 timed out)
2015-11-03 04:02:49.482052 Server/RaftConsensus.cc:2846 in printElectionState() NOTICE[2:startNewElection]: server=2, term=2113, state=CANDIDATE, leader=0, vote=2
2015-11-03 04:02:49.482072 Storage/SegmentedLog.cc:778 in updateMetadata() NOTICE[2:startNewElection]: Writing new storage metadata (version 2219) to metadata1
2015-11-03 04:02:54.895499 Server/RaftConsensus.cc:282 in callRPC() WARNING[2:Peer(3)]: RPC to server succeeded after 70 failures
2015-11-03 04:02:54.895525 Server/RaftConsensus.cc:2808 in requestVote() NOTICE[2:Peer(3)]: Vote denied by server 3 for term 2113
2015-11-03 04:02:54.895619 Server/RaftConsensus.cc:282 in callRPC() WARNING[2:Peer(1)]: RPC to server succeeded after 23 failures
2015-11-03 04:02:54.895652 Server/RaftConsensus.cc:2808 in requestVote() NOTICE[2:Peer(1)]: Vote denied by server 1 for term 2113
2015-11-03 04:02:55.152641 Server/RaftConsensus.cc:2867 in startNewElection() NOTICE[2:startNewElection]: Running for election in term 2114 (previous candidacy for term 2113 timed out)
2015-11-03 04:02:55.152666 Server/RaftConsensus.cc:2846 in printElectionState() NOTICE[2:startNewElection]: server=2, term=2114, state=CANDIDATE, leader=0, vote=2
2015-11-03 04:02:55.152706 Storage/SegmentedLog.cc:778 in updateMetadata() NOTICE[2:startNewElection]: Writing new storage metadata (version 2220) to metadata2
2015-11-03 04:02:55.161240 Server/RaftConsensus.cc:2803 in requestVote() NOTICE[2:Peer(3)]: Got vote from server 3 for term 2114
2015-11-03 04:02:55.161267 Server/RaftConsensus.cc:2490 in becomeLeader() NOTICE[2:Peer(3)]: Now leader for term 2114 (appending no-op at index 16629)
2015-11-03 04:02:55.161274 Server/RaftConsensus.cc:2846 in printElectionState() NOTICE[2:Peer(3)]: server=2, term=2114, state=LEADER,    leader=2, vote=2
2015-11-03 04:03:27.433844 Server/RaftConsensus.cc:2154 in stepDownThreadMain() NOTICE[2:stepDown]: No broadcast for a timeout, stepping down from leader of term 2114 (converting to follower in term 2115)
2015-11-03 04:03:27.433876 Storage/SegmentedLog.cc:778 in updateMetadata() NOTICE[2:stepDown]: Writing new storage metadata (version 2221) to metadata1
2015-11-03 04:03:27.438944 Server/RaftConsensus.cc:2846 in printElectionState() NOTICE[2:stepDown]: server=2, term=2115, state=FOLLOWER,  leader=0, vote=0
2015-11-03 04:03:35.164613 Server/RaftConsensus.cc:2870 in startNewElection() NOTICE[2:startNewElection]: Running for election in term 2116
2015-11-03 04:03:35.164638 Server/RaftConsensus.cc:2846 in printElectionState() NOTICE[2:startNewElection]: server=2, term=2116, state=CANDIDATE, leader=0, vote=2
2015-11-03 04:03:35.164659 Storage/SegmentedLog.cc:778 in updateMetadata() NOTICE[2:startNewElection]: Writing new storage metadata (version 2222) to metadata2
2015-11-03 04:03:39.378562 Server/ServerStats.cc:148 in dumpToDebugLog() NOTICE[2:StatsDumper]: ServerStats:
server_id: 2
addresses: "192.168.8.34"
start_at: 1446523419378406286
end_at: 1446523419378450694
raft {
  current_term: 2116
  state: CANDIDATE
  commit_index: 16689
  last_log_index: 16690
  leader_id: 0
  voted_for: 2
  start_election_at: 1446523422117357032
  withhold_votes_until: -7776880372783699812
  cluster_time: 15616398256510
  cluster_time_epoch: 15603272756102
  last_snapshot_index: 0
  last_snapshot_bytes: 0
  log_start_index: 1
  log_bytes: 3465222
  last_snapshot_term: 0
  last_snapshot_cluster_time: 0
  num_entries_truncated: 19
  peer {
    server_id: 1
    addresses: "192.168.8.33"
    old_member: true
    new_member: false
    staging_member: false
    request_vote_done: false
    have_vote: false
    backoff_until: 1446523405933463909
  }
  peer {
    server_id: 2
    addresses: "192.168.8.34"
    old_member: true
    new_member: false
    staging_member: false
  }
  peer {
    server_id: 3
    addresses: "192.168.8.35"
    old_member: true
    new_member: false
    staging_member: false
    request_vote_done: false
    have_vote: false
    backoff_until: 1446523405933213282
  }
}
storage {
  num_segments: 20
  open_segment_bytes: 108881
  metadata_version: 2222
  metadata_write_nanos {
    average: 3695109.9090909092
    count: 2222
    ewma2: 3788212.214560519
    ewma4: 3723413.8854548456
    exceptional_count: 0
    last: 3480358
    min: 322927
    max: 62131317
    sum: 8210534218
    stddev: 3977679.6864882521
  }
  filesystem_ops_nanos {
    average: 790993.54927120963
    count: 16054
    ewma2: 1244647.6260738196
    ewma4: 771464.127916003
    exceptional_count: 0
    last: 2199668
    min: 167693
    max: 158011023
    sum: 12698610440
    stddev: 3143156.3835586049
  }
}
state_machine {
  snapshotting: false
  last_applied: 16689
  num_sessions: 3
  num_snapshots_attempted: 0
  num_snapshots_failed: 0
  num_redundant_advance_version_entries: 2
  num_rejected_advance_version_entries: 0
  num_successful_advance_version_entries: 1
  num_total_advance_version_entries: 3
  min_supported_version: 1
  max_supported_version: 2
  running_version: 2
  tree {
    num_conditions_checked: 15263
    num_conditions_failed: 1216
    num_make_directory_attempted: 21
    num_make_directory_success: 21
    num_list_directory_attempted: 19218
    num_list_directory_success: 19218
    num_remove_directory_attempted: 0
    num_remove_directory_parent_not_found: 0
    num_remove_directory_target_not_found: 0
    num_remove_directory_done: 0
    num_remove_directory_success: 0
    num_write_attempted: 15209
    num_write_success: 15209
    num_read_attempted: 70312
    num_read_success: 70094
    num_remove_file_attempted: 0
    num_remove_file_parent_not_found: 0
    num_remove_file_target_not_found: 0
    num_remove_file_done: 0
    num_remove_file_success: 0
  }
  num_unknown_requests: 0
  may_snapshot_at: -7776880372783699936
}

2015-11-03 04:03:40.168601 Server/RaftConsensus.cc:294 in callRPC() WARNING[2:Peer(1)]: RPC to server failed: Server 192.168.8.33 (resolved to 192.168.8.33:5254) timed out
2015-11-03 04:03:40.168685 Server/RaftConsensus.cc:294 in callRPC() WARNING[2:Peer(3)]: RPC to server failed: Server 192.168.8.35 (resolved to 192.168.8.35:5254) timed out
2015-11-03 04:03:40.445678 RPC/MessageSocket.cc:43 in dupOrPanic() ERROR[2:evloop]: Failed to dup(1023): Too many open files Exiting...
@ongardie ongardie added the bug label Nov 6, 2015
@ongardie
Copy link
Member

ongardie commented Nov 6, 2015

Fun. I hadn't really worried about the number of open file descriptors in the past, and I can't rule out a leak somewhere.

My first worry was that maybe the SegmentedLog had gotten in some pathological situation and generated a bunch of small segments, but it's only got 20, so that's not it:

storage {
  num_segments: 20
  ...
  metadata_version: 2222

The other good news is that your state machine has never tried to take a snapshot, so that can't be it:

state_machine {
  snapshotting: false
  ...
  num_snapshots_attempted: 0

It might be a leak. It might be sockets, epoll fds, timerfds, signalfds, or other non-obvious stuff. It's hard to say with the current information.

If you can repro this (it sounds like you can), try to list out what's in /proc/<pid>/fd. Here's one from a healthy idle server. Something like this would probably narrow it down pretty quickly:

~$ ls -l /proc/$(pidof LogCabin)/fd
total 0
lrwx------ 1 ongardie ongardie 64 Nov  6 10:28 0 -> /dev/pts/4
lrwx------ 1 ongardie ongardie 64 Nov  6 10:28 1 -> /dev/pts/4
lr-x------ 1 ongardie ongardie 64 Nov  6 10:28 10 -> /home/ongardie/logcabin/storage/server1
lr-x------ 1 ongardie ongardie 64 Nov  6 10:28 11 -> /home/ongardie/logcabin/storage/server1/lock
lr-x------ 1 ongardie ongardie 64 Nov  6 10:28 12 -> /home/ongardie/logcabin/storage/server1/log
lr-x------ 1 ongardie ongardie 64 Nov  6 10:28 13 -> /home/ongardie/logcabin/storage/server1/snapshot
lr-x------ 1 ongardie ongardie 64 Nov  6 10:28 14 -> /home/ongardie/logcabin/storage/server1/log/Segmented-Binary
lrwx------ 1 ongardie ongardie 64 Nov  6 10:28 15 -> /home/ongardie/logcabin/storage/server1/log/Segmented-Binary/open-1
lrwx------ 1 ongardie ongardie 64 Nov  6 10:28 16 -> /home/ongardie/logcabin/storage/server1/log/Segmented-Binary/open-2
lrwx------ 1 ongardie ongardie 64 Nov  6 10:28 19 -> /home/ongardie/logcabin/storage/server1/log/Segmented-Binary/open-3
lrwx------ 1 ongardie ongardie 64 Nov  6 10:28 2 -> /dev/pts/4
lrwx------ 1 ongardie ongardie 64 Nov  6 10:28 20 -> anon_inode:[signalfd]
lrwx------ 1 ongardie ongardie 64 Nov  6 10:28 21 -> /home/ongardie/logcabin/storage/server1/log/Segmented-Binary/open-4
lrwx------ 1 ongardie ongardie 64 Nov  6 10:28 3 -> anon_inode:[eventpoll]
lrwx------ 1 ongardie ongardie 64 Nov  6 10:28 4 -> anon_inode:[timerfd]
lrwx------ 1 ongardie ongardie 64 Nov  6 10:28 5 -> anon_inode:[signalfd]
lrwx------ 1 ongardie ongardie 64 Nov  6 10:28 6 -> anon_inode:[signalfd]
lrwx------ 1 ongardie ongardie 64 Nov  6 10:28 7 -> anon_inode:[signalfd]
lrwx------ 1 ongardie ongardie 64 Nov  6 10:28 8 -> socket:[14558690]
lr-x------ 1 ongardie ongardie 64 Nov  6 10:28 9 -> /home/ongardie/logcabin/storage

@ongardie
Copy link
Member

@nhardt, what's the latest on this one?

@nhardt
Copy link
Contributor Author

nhardt commented Mar 15, 2016

Nothing new. Likely still occurring but as there is minimal impact (vs other pressing issue) I have not tracked it down.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ongardie @nhardt