Intermittent issue in test/ipc_tests.cpp Fatal glibc error: pthread_mutex_lock.c:450 (__pthread_mutex_lock_full): assertion failed: e != ESRCH || !robust

[maflcko]

Happened on current master d29fc3a in ci_i686_multiprocess

Running tests: wallet_util_tests from wallet/test/rpc_util_tests.cpp
Running tests: scriptpubkeyman_tests from wallet/test/scriptpubkeyman_tests.cpp
Running tests: walletload_tests from wallet/test/walletload_tests.cpp
Running tests: group_outputs_tests from wallet/test/group_outputs_tests.cpp
Running tests: db_tests from wallet/test/db_tests.cpp
Running tests: ipc_tests from test/ipc_tests.cpp
/bin/bash: line 2: 28702 Aborted                 (core dumped) test/test_bitcoin --catch_system_errors=no -l test_suite -t "$(cat test/ipc_tests.cpp | grep -E "(BOOST_FIXTURE_TEST_SUITE\\(|BOOST_AUTO_TEST_SUITE\\()" | cut -d '(' -f 2 | cut -d ',' -f 1 | cut -d ')' -f 1)" -- DEBUG_LOG_OUT > "$TEST_LOGFILE" 2>&1
Running 1 test case...
Test cases order is shuffled using seed: 227983234
Entering test module "Bitcoin Core Test Suite"
wallet/test/feebumper_tests.cpp(18): Test suite "feebumper_tests" is skipped because disabled
test/validation_flush_tests.cpp(13): Test suite "validation_flush_tests" is skipped because disabled
test/raii_event_tests.cpp(16): Test suite "raii_event_tests" is skipped because disabled
test/denialofservice_tests.cpp(36): Test suite "denialofservice_tests" is skipped because disabled
test/i2p_tests.cpp(45): Test suite "i2p_tests" is skipped because disabled
test/interfaces_tests.cpp(16): Test suite "interfaces_tests" is skipped because disabled
test/bloom_tests.cpp(25): Test suite "bloom_tests" is skipped because disabled
test/txvalidation_tests.cpp(20): Test suite "txvalidation_tests" is skipped because disabled
test/orphanage_tests.cpp(19): Test suite "orphanage_tests" is skipped because disabled
test/blockencodings_tests.cpp(19): Test suite "blockencodings_tests" is skipped because disabled
test/pmt_tests.cpp(28): Test suite "pmt_tests" is skipped because disabled
test/streams_tests.cpp(15): Test suite "streams_tests" is skipped because disabled
test/skiplist_tests.cpp(15): Test suite "skiplist_tests" is skipped because disabled
test/coinstatsindex_tests.cpp(16): Test suite "coinstatsindex_tests" is skipped because disabled
test/minisketch_tests.cpp(17): Test suite "minisketch_tests" is skipped because disabled
test/allocator_tests.cpp(16): Test suite "allocator_tests" is skipped because disabled
test/mempool_tests.cpp(16): Test suite "mempool_tests" is skipped because disabled
test/coins_tests.cpp(100): Test suite "coins_tests" is skipped because disabled
test/hash_tests.cpp(14): Test suite "hash_tests" is skipped because disabled
test/uint256_tests.cpp(17): Test suite "uint256_tests" is skipped because disabled
wallet/test/coinselector_tests.cpp(24): Test suite "coinselector_tests" is skipped because disabled
test/blockfilter_index_tests.cpp(25): Test suite "blockfilter_index_tests" is skipped because disabled
test/validation_chainstate_tests.cpp(21): Test suite "validation_chainstate_tests" is skipped because disabled
test/disconnected_transactions.cpp(10): Test suite "disconnected_transactions" is skipped because disabled
test/sync_tests.cpp(80): Test suite "sync_tests" is skipped because disabled
test/getarg_tests.cpp(19): Test suite "getarg_tests" is skipped because disabled
test/logging_tests.cpp(20): Test suite "logging_tests" is skipped because disabled
test/blockmanager_tests.cpp(25): Test suite "blockmanager_tests" is skipped because disabled
test/ipc_tests.cpp(8): Entering test suite "ipc_tests"
test/ipc_tests.cpp(9): Entering test case "ipc_tests"
Fatal glibc error: pthread_mutex_lock.c:450 (__pthread_mutex_lock_full): assertion failed: e != ESRCH || !robust
make[3]: *** [Makefile:22535: test/ipc_tests.cpp.test] Error 1
make[3]: *** Waiting for unfinished jobs....
make[3]: Leaving directory '/ci_container_base/ci/scratch/build/bitcoin-i686-pc-linux-gnu/src'
make[2]: *** [Makefile:20499: check-am] Error 2
make[2]: Leaving directory '/ci_container_base/ci/scratch/build/bitcoin-i686-pc-linux-gnu/src'
make[1]: *** [Makefile:20164: check-recursive] Error 1
make[1]: Leaving directory '/ci_container_base/ci/scratch/build/bitcoin-i686-pc-linux-gnu/src'
make: *** [Makefile:755: check-recursive] Error 1

[maflcko]

# podman exec ci_i686_multiprocess  uname -a
Linux a943c4649ecd 5.14.21-150400.24.100-default #1 SMP PREEMPT_DYNAMIC Mon Dec 4 19:12:13 UTC 2023 (3f5cd84) x86_64 x86_64 x86_64 GNU/Linux

[ryanofsky]

I wonder if there's any way to get a stack trace. Seeing log output could also be helpful as the test should be logging everything with LogPrintf.

[maflcko]

I threw away the pod, so the logs are gone, but I wonder why they were not printed to the screen.

Otherwise, if this issue happens on Cirrus or GHA, the logs won't be available either.

[maflcko]

Actually, I think the logs are complete for the log level test_suite. If more details are needed test_suite could be changed to all.

[maflcko]

stack trace

#29900

[maflcko]

I guess it could make more sense if someone tried to reproduce this in rr.