How to resolve backtrace
Seastar applications will dump backtrace to stderr on various occasions, e.g. on abort or SIGSEGV. The backtrace may look like this:
Aborting on shard 0.
Backtrace:
0x2106d693
0x21e2019e
0x226efa48
0x2288aa34
0x227ca6da
0x21e92210
0x21e933a7
0x21f7b11d
0x21f7bf7d
0x21f7cdc9
0x21f7d4f0
0x21e82b56
0x220b7d5d
0x220b7ead
0x220b7fd8
0x2264a560
0x2264a9ee
0x20925b7b
0x2092924e
0x2092e343
0x20fdbb82
0x218e6c39
/lib/x86_64-linux-gnu/libc.so.6+0x203f0
0x208e1589
To resolve the backtrace, you need an un-stripped Scylla binary matching your version. If unsure if your binary is un-stripped, use file command to determine that:
$ file build/release/scylla
build/release/scylla: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=bf11ae9efddedcc451606140d8fdfa69005b4076, not stripped
There should be not stripped in the output.
[scylla-test@longevity-1-7-gce-amosbranch-scylla-db-node-50f18619-001 ~]$ file /usr/bin/scylla
/usr/bin/scylla: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=a3bdc1b0a09a906facf2021fdafdec8809d999f2, stripped
[scylla-test@longevity-1-7-gce-amosbranch-scylla-db-node-50f18619-001 ~]$ file /usr/lib/debug/usr/bin/scylla.debug
/usr/lib/debug/usr/bin/scylla.debug: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=a3bdc1b0a09a906facf2021fdafdec8809d999f2, not stripped
If your binary is stripped, you need to unstrip it. Follow instrucitons in the What if I have a stripped binary secton below.
scylla-debuginfo should be installed in production environment, scylla.debug is not stripped.
For ubuntu 14.04 use sudo apt-get install scylla-server-dbg
Starting from version 2.1 scylla uses ASLR friendly backtraces, that is
instead of printing absolute addressess the module-relative address is
printed. To convert backtraces to symbols, run the seastar-addr2line
found in the scripts directory of the seastar git repository.
$ seastar-addr2line -e /usr/bin/scylla
0x2106d693
0x21e2019e
0x226efa48
0x2288aa34
0x227ca6da
0x21e92210
0x21e933a7
0x21f7b11d
0x21f7bf7d
0x21f7cdc9
0x21f7d4f0
0x21e82b56
0x220b7d5d
0x220b7ead
0x220b7fd8
0x2264a560
0x2264a9ee
0x20925b7b
0x2092924e
0x2092e343
0x20fdbb82
0x218e6c39
/lib/x86_64-linux-gnu/libc.so.6+0x203f0
0x208e1589
For ubuntu 14.04
- sudo apt-get install binutils
- sudo apt-get install binutils-multiarch
You may need to press Ctrl+D after pasting the addresses to the input of the script to flush it's stdin buffers.
It should print an output like this:
seastar::frame& boost::container::vector<seastar::frame, boost::container::container_detail::static_storage_allocator<seastar::frame, 64ul> >::emplace_back<seastar::frame>(seastar::frame&&) at /usr/include/boost/container/vector.hpp:1744
(inlined by) operator() at /home/botond/Workspace/scylla/seastar/util/backtrace.cc:86
column_family::column_family(seastar::lw_shared_ptr<schema const>, column_family::config, db::commitlog*, compaction_manager&, cell_locker_stats&) at /home/botond/Workspace/scylla/database.cc:162
column_family::config::config(column_family::config&&) at /home/botond/Workspace/scylla/database.hh:288 (discriminator 4)
std::pair<seastar::basic_sstring<char, unsigned int, 15u>, seastar::lw_shared_ptr<query::result_set> >&& std::forward<std::pair<seastar::basic_sstring<char, unsigned int, 15u>, seastar::lw_shared_ptr<query::result_set> >&&>(std::remove_reference<std::pair<seastar::basic_sstring<char, unsigned int, 15u>, seastar::lw_shared_ptr<query::result_set> >&&>::type&) at /usr/include/c++/6/bits/move.h:77
seastar::lw_shared_ptr<schema const>::operator=(seastar::lw_shared_ptr<schema const>&&) at /home/botond/Workspace/scylla/./seastar/core/shared_ptr.hh:292
database::add_column_family(keyspace&, seastar::lw_shared_ptr<schema const>, column_family::config) at /home/botond/Workspace/scylla/database.cc:2755
database::add_column_family(keyspace&, seastar::lw_shared_ptr<schema const>, column_family::config) at /home/botond/Workspace/scylla/database.cc:2753
...
It's also possible to pass the addresses as arguments to the seastar-addr2line script, like this:
$ addr2line -e /usr/bin/scylla 0x2106d693 0x21e2019e 0x226efa48 0x2288aa34 0x227ca6da
seastar-addr2line can also read the backtraces from a file:
$ addr2line -e /usr/bin/scylla -f backtrace.txt
For releases older than 2.1 the absolute addresses are printed. Resolving these is done with addr2line directly, like this:
$ addr2line -Cfpi -e /usr/bin/scylla
0x00000000004893ba
0x000000000048949b
0x00007f0f806f4a00
0x00007f0f80357a28
0x00007f0f8035962a
0x000000000051acd4
0x000000000051ad50
0x000000000051aed3
0x000000000079b295
0x0000000000839668
0x00000000008396a9
0x0000000000c53490
0x0000000000c8bafc
0x0000000000c55915
0x0000000000c575f5
0x0000000000acb656
0x0000000000acbc88
0x000000000048c22f
0x00000000004c6b83
0x0000000000543df4
0x000000000041deab
0x00007f0f80343580
0x0000000000487819
Scylla <= 2.0 also uses single-line backtraces as well. To resolve a single-line backtrace, pass it as addr2line arguments, like this:
$ addr2line -Cfpi -e /usr/bin/scylla 0x4ec903, 0x4ec961, 0x57b87f, 0x82d529, 0x4e1d6f, 0x5317d2, 0x533010, 0x58e93d, 0x7f9cae6a7dc4, 0x7f9cae3d673c
If you see stripped in the output of file then you need to unstrip it in order to decode the backtrace.
First, fetch the debug info file for your binary. See this page on how to do that.
The debug info file should get installed in /usr/lib/debug/.build-id/${x}/${y}.debug where ${x} and ${y} form the BuildID of the binary. You can find BuildID in the output of file command.
For example, if BuildID is bf11ae9efddedcc451606140d8fdfa69005b4076, then the debug info is in
/usr/lib/debug/.build-id/bf/11ae9efddedcc451606140d8fdfa69005b4076.debug
Then run eu-unstrip like this:
eu-unstrip /usr/bin/scylla /usr/lib/debug/.build-id/bf/11ae9efddedcc451606140d8fdfa69005b4076.debug
It will merge the binary with the debug info file and put the result in the debug info file. After that, you can use /usr/lib/debug/.build-id/bf/11ae9efddedcc451606140d8fdfa69005b4076.debug to resolve symbols instead of /usr/bin/scylla.
Test binaries are stripped by default and --test-debuginfo must be passed to configure in order to get them unstripped.