Seastar heap profiler

Building faster libunwind on older Fedora

The default libunwind 8.0.1 which comes with Fedora 23 has a performance problem of calling sigprocmask on every context creation. There is a way to work it around.

git clone git://git.sv.gnu.org/libunwind.git
cd libunwind
dnf install autoconf automake libtool 
./autogen.sh
./configure  --enable-block-signals=no
make
sudo make install

Make sure that scylla uses the right library version (use ldd build/release/scylla), e.g. like this:

sudo cp ./src/.libs/libunwind.so.8.0.1 /lib64/libunwind.so.8.0.1

Enabling profile collection

Build seastar with heap profiler support:

./configure.py --cflags=-DSEASTAR_HEAPPROF

NOTE: If you want to avoid recompilation of the whole tree, you can insert instead the following line at the top of seastar/core/memory.cc and seastar/core/reactor.cc:

#define SEASTAR_HEAPPROF

Then pass --heapprof to command line arguments of your seastar application to enable profiling.

If you want to profile only certain section of the code, enable profiling by calling:

seastar::memory::set_heap_profiling_enabled(true);
//...
seastar::memory::set_heap_profiling_enabled(false);

Install FlameGraph

On recent Fedora versions FlameGraph is available from the system repositories:

dnf install flamegraph.noarch

You can also use the git repository directly:

git clone https://github.com/brendangregg/FlameGraph.git
export PATH=$PATH:`pwd`/FlameGraph

Generating flamegraphs

Dump the profile of current shard using GDB:

(gdb) source scylla-gdb.py
(gdb) scylla heapprof --flame

Generate flamegraph from heapprof.stacks:

$ flamegraph.pl --colors mem < heapprof.stacks > heapprof.svg

Note that not all image viewer applications will make the svg zoomable. For best result use your browser to open it.