Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Missing debug symbols for perf analysis on process inside docker container #513

Open
BartVandewoestyne opened this issue Sep 27, 2023 · 7 comments
Open

Missing debug symbols for perf analysis on process inside docker container #513

BartVandewoestyne opened this issue Sep 27, 2023 · 7 comments
Labels
question

Comments

@BartVandewoestyne
Copy link

It is very likely that this is not a bug, but rather a problem on my side. As I did not find a forum or mailing list to ask my question, I emailed Milian Wolff and he kindly suggested me to post my problem as an issue on GitHub, hence why it is posted here.

I would like to do some performance analysis using hotspot on a process that is running inside a Docker container. I collect my perf data on the host (not in the docker container!) as follows:

sudo perf record --call-graph dwarf -p <pid-on-host>

where <pid-on-host> is the process ID on the host of the process I'm interested in (the process is /usr/sbin/sabine inside the docker container and there it has another PID).

Using docker ps to find the ID of my docker container, and then docker inspect as follows:

bvandewoestyne@BVANDEW-70TZQ73:~/projects/virtual_ngp_network/docker_files$ docker inspect 9347ec238188 | grep MergedDir
                "MergedDir": "/var/lib/docker/overlay2/0cf422ea6e341c992254d2221c77f3211629e546b44c9a46cf2e9b01654bc38a/merged",

I search for the directory that has the same contents as the root filesystem of my docker container, so that I can pass that to the --sysroot option of hotspot so that all symbols can be found.

Then, I start hotspot with the --sysroot option as follows:

sudo /home/bvandewoestyne/bin/hotspot-v1.4.1-x86_64.AppImage --sysroot=/var/lib/docker/overlay2/0cf422ea6e341c992254d2221c77f3211629e546b44c9a46cf2e9b01654bc38a/merged perf.data

I need sudo to access the merged directory.

I get the following errors in my terminal:

bvandewoestyne@BVANDEW-70TZQ73:~/projects/virtual_ngp_network/docker_files$ sudo /home/bvandewoestyne/bin/hotspot-v1.4.1-x86_64.AppImage --sysroot=/var/lib/docker/overlay2/0cf422ea6e341c992254d2221c77f3211629e546b44c9a46cf2e9b01654bc38a/merged perf.data
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
kf.coreaddons: "Could not find plugin konsolepart"
unhandled feature PerfHeader::MEM_TOPOLOGY 88
unhandled feature PerfHeader::SAMPLE_TIME 16
unhandled feature PerfHeader::CACHE 4188
unhandled feature PerfHeader::BPF_BTF 4
unhandled feature PerfHeader::BPF_PROG_INFO 2500
unhandled feature PerfHeader::CPU_PMU_CAPS 412
feature not properly read PerfHeader::CPU_TOPOLOGY 700 484
Linux version "5.15.116" detected. Switching to automatic buffering.
unhandled event type 79   PERF_RECORD_TIME_CONV
unhandled event type 73   PERF_RECORD_THREAD_MAP
unhandled event type 74   PERF_RECORD_CPU_MAP
unhandled event type 17   PERF_RECORD_KSYMBOL
unhandled event type 18   PERF_RECORD_BPF_EVENT
unhandled event type 17   PERF_RECORD_KSYMBOL
unhandled event type 18   PERF_RECORD_BPF_EVENT
unhandled event type 17   PERF_RECORD_KSYMBOL
unhandled event type 18   PERF_RECORD_BPF_EVENT
unhandled event type 17   PERF_RECORD_KSYMBOL
unhandled event type 18   PERF_RECORD_BPF_EVENT
unhandled event type 17   PERF_RECORD_KSYMBOL
unhandled event type 18   PERF_RECORD_BPF_EVENT
unhandled event type 17   PERF_RECORD_KSYMBOL
unhandled event type 18   PERF_RECORD_BPF_EVENT
unhandled event type 17   PERF_RECORD_KSYMBOL
unhandled event type 18   PERF_RECORD_BPF_EVENT
unhandled event type 17   PERF_RECORD_KSYMBOL
unhandled event type 18   PERF_RECORD_BPF_EVENT
No DWARF information found 1
No DWARF information found 2
No DWARF information found 2
No DWARF information found 2

and more importantly, in hotspot I see that for sabine I have lots of missing debug symbols and I see a lot of question marks in the flamegraph:
hotspot_gui_missing_symbols
hotspot_question_marks_for_sabine

For as far as I know, my sabine binary has debug info in it, as can be seen with file (notice the with debug_info in the output):

bvandewoestyne@BVANDEW-70TZQ73:~/projects/virtual_ngp_network/docker_files$ sudo file /var/lib/docker/overlay2/0cf422ea6e341c992254d2221c77f3211629e546b44c9a46cf2e9b01654bc38a/merged/usr/sbin/sabine
/var/lib/docker/overlay2/0cf422ea6e341c992254d2221c77f3211629e546b44c9a46cf2e9b01654bc38a/merged/usr/sbin/sabine: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d, with debug_info, not stripped

Would you have any idea why I still have these missing debug symbols in hotspot? Or any idea how to diagnose this further?

Some info of my host system:

bvandewoestyne@BVANDEW-70TZQ73:~$ uname -a
Linux BVANDEW-70TZQ73 5.15.0-84-generic #93~20.04.1-Ubuntu SMP Wed Sep 6 16:15:40 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

bvandewoestyne@BVANDEW-70TZQ73:~$ cat /etc/issue.net 
Ubuntu 20.04.6 LTS

bvandewoestyne@BVANDEW-70TZQ73:~$ perf --version
perf version 5.15.116

bvandewoestyne@BVANDEW-70TZQ73:~$ hotspot-v1.4.1-x86_64.AppImage --version
hotspot 1.4.1

Kind regards,
Bart

PS: a similar question has been asked here (but not answered): https://stackoverflow.com/questions/71685770/profile-a-process-inside-a-container-from-host-using-perf
@milianw
Copy link
Member

milianw commented Sep 28, 2023

I definitely have done something along those lines some years ago, but I don't have that setup available to me anymore so I don't know for sure what steps I did.

But debugging this should be relatively straight forward:

sudo strace -e file -f /home/bvandewoestyne/bin/hotspot-v1.4.1-x86_64.AppImage --sysroot=/var/lib/docker/overlay2/0cf422ea6e341c992254d2221c77f3211629e546b44c9a46cf2e9b01654bc38a/merged perf.data |& grep sabine

This should show you where hotspot or perfparser try to access the sabine binary. Please paste it here so we can check what's happening.

Also, to cross-check: You can setup docker to allow launching graphical applications therein. Can you try to do that and run the AppImage within the docker container to check whether that works better? It shouldn't be needed, but it would give us a second view into this problem.

@milianw milianw added question and removed bug labels Sep 28, 2023
@BartVandewoestyne
Copy link
Author

Should the GUI popup fast under that strace command? Because when I run it, I don't get any grep output. After waiting for 5 minutes, I was still stuck at

$ sudo strace -e file -f /home/bvandewoestyne/bin/hotspot-v1.4.1-x86_64.AppImage --sysroot=/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged perf.data |& grep sabine

without any output. I see the following processes are running:

bvandewoestyne@BVANDEW-70TZQ73:~$ ps aux | grep hotspot
root      489142  0.0  0.0  38604  7656 pts/1    S+   13:10   0:00 sudo strace -e file -f /home/bvandewoestyne/bin/hotspot-v1.4.1-x86_64.AppImage --sysroot=/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged perf.data
root      489144  0.0  0.0  10736  3740 pts/1    D+   13:10   0:00 strace -e file -f /home/bvandewoestyne/bin/hotspot-v1.4.1-x86_64.AppImage --sysroot=/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged perf.data
root      489149  0.1  0.0  12396  2288 ?        tsl  13:10   0:00 /home/bvandewoestyne/bin/hotspot-v1.4.1-x86_64.AppImage --sysroot=/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged perf.data
root      489155  0.0  0.0 180796 17968 pts/1    t+   13:10   0:00 /tmp/.mount_hotspoEaiKtc/usr/bin/hotspot --sysroot=/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged perf.data
bvandew+  492873  0.0  0.0   9380  2760 pts/3    S+   13:13   0:00 grep --color=auto hotspot

@BartVandewoestyne
Copy link
Author

I was able to successfully grab the strace with the hotspot installed via apt on my Ubuntu:

bvandewoestyne@BVANDEW-70TZQ73:~/projects/virtual_ngp_network/docker_files$ cat /etc/issue.net 
Ubuntu 20.04.6 LTS
bvandewoestyne@BVANDEW-70TZQ73:~/projects/virtual_ngp_network/docker_files$ which hotspot
/usr/bin/hotspot
bvandewoestyne@BVANDEW-70TZQ73:~/projects/virtual_ngp_network/docker_files$ hotspot --version
hotspot 1.1.0

I was able to strace with the command:

$ sudo strace -e file -f hotspot --sysroot=/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged perf.data |& grep sabine | xclip -selection clipboard

for which the output was:

[pid 560271] statx(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug//usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7fffbfeac220) = -1 ENOENT (No such file or directory)
[pid 560271] statx(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7fffbfeac220) = -1 ENOENT (No such file or directory)
[pid 560271] statx(AT_FDCWD, "/root/.debug//usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_ALL|0x1000, stx_attributes=0, stx_mode=S_IFDIR|0755, stx_size=4096, ...}) = 0
[pid 560271] statx(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_ALL|0x1000, stx_attributes=0, stx_mode=S_IFREG|0755, stx_size=9104232, ...}) = 0
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = 5
[pid 560271] statx(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug//usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7fffbfeac220) = -1 ENOENT (No such file or directory)
[pid 560271] statx(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7fffbfeac220) = -1 ENOENT (No such file or directory)
[pid 560271] statx(AT_FDCWD, "/root/.debug//usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_ALL|0x1000, stx_attributes=0, stx_mode=S_IFDIR|0755, stx_size=4096, ...}) = 0
[pid 560271] statx(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_ALL|0x1000, stx_attributes=0, stx_mode=S_IFREG|0755, stx_size=9104232, ...}) = 0
[pid 560271] statx(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug//usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7fffbfeac220) = -1 ENOENT (No such file or directory)
[pid 560271] statx(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7fffbfeac220) = -1 ENOENT (No such file or directory)
[pid 560271] statx(AT_FDCWD, "/root/.debug//usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_ALL|0x1000, stx_attributes=0, stx_mode=S_IFDIR|0755, stx_size=4096, ...}) = 0
[pid 560271] statx(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_ALL|0x1000, stx_attributes=0, stx_mode=S_IFREG|0755, stx_size=9104232, ...}) = 0
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = 8
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = 9
[pid 560271] openat(AT_FDCWD, "/root/.debug/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/.debug/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/.debug/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = 9
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOTDIR (Not a directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOTDIR (Not a directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOTDIR (Not a directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOTDIR (Not a directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] lstat("/root/.debug/usr/sbin/sabine", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 560271] lstat("/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 560271] lstat("/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", {st_mode=S_IFREG|0755, st_size=9104232, ...}) = 0
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = 9
[pid 560271] openat(AT_FDCWD, "/root/.debug/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/.debug/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/.debug/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = 9
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOTDIR (Not a directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOTDIR (Not a directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOTDIR (Not a directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOTDIR (Not a directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] lstat("/root/.debug/usr/sbin/sabine", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 560271] lstat("/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 560271] lstat("/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", {st_mode=S_IFREG|0755, st_size=9104232, ...}) = 0
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = 6
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = 8
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = 10
[pid 560271] openat(AT_FDCWD, "/root/.debug/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/.debug/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/.debug/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = 10
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOTDIR (Not a directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOTDIR (Not a directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOTDIR (Not a directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOTDIR (Not a directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] lstat("/root/.debug/usr/sbin/sabine", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 560271] lstat("/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 560271] lstat("/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", {st_mode=S_IFREG|0755, st_size=9104232, ...}) = 0
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/lib/debug/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = 10
[pid 560271] openat(AT_FDCWD, "/root/.debug/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/.debug/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/.debug/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = 10
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOTDIR (Not a directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOTDIR (Not a directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOTDIR (Not a directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOTDIR (Not a directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] openat(AT_FDCWD, "/var/lib/docker/overlay2/86aa800d14f4efdfabc0110938bf67ef8c9697423bc8a4b8168ad68d7bb3062e/merged/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 560271] lstat("/root/.debug/usr/sbin/sabine", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 560271] lstat("/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 560271] lstat("/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", {st_mode=S_IFREG|0755, st_size=9104232, ...}) = 0

Does that give any insight?

@milianw
Copy link
Member

milianw commented Sep 28, 2023

Yes, it finds some cached data over at

[pid 560271] lstat("/root/.debug/usr/sbin/sabine", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0

which is probably wrong for the given buildid?

You can try to workaround this by passing an explicit --debugPaths and/or --appPath to hotspot which points into your sysroot.

But ideally you also test the latest appimage (see e.g. https://github.com/KDAB/hotspot/actions/runs/6335963163 ) and see if that one works better. If not, then there's clearly a bug here - we shouldn't try to load this file outside the sysroot imo.

@BartVandewoestyne
Copy link
Author

Using the latest AppImage hotspot-v1.4.1-175-gfaf33ed-x86_64.AppImage, when I run with only the --sysroot option as

sudo ~/bin/hotspot-v1.4.1-175-gfaf33ed-x86_64.AppImage --sysroot=/var/lib/docker/overlay2/20dbe9864fa8531dc0c27929dd78e7112aa0edba84e339b99722a495c40233ff/merged perf.data

then I get the following parse errors:
hotspot_with_sysroot
Notice that for sabine the symbols were not found.

When I run with both the --sysroot and --debugPath options as

sudo ~/bin/hotspot-v1.4.1-175-gfaf33ed-x86_64.AppImage --sysroot=/var/lib/docker/overlay2/20dbe9864fa8531dc0c27929dd78e7112aa0edba84e339b99722a495c40233ff/merged --debugPaths=/var/lib/docker/overlay2/20dbe9864fa8531dc0c27929dd78e7112aa0edba84e339b99722a495c40233ff/merged perf.data

Then the sabine symbols are found, which is already good, but apparently then the "kernel symbol mapping file" cannot be found/parsed:
hotspot_with_sysroot_and_debugPaths

Important sidenote: colleagues of me have done the same test (on both Ubuntu 20 and 22) and for them the sabine symbols were found without using the --debugPaths option, only using the --sysroot option... which still puzzles me...

So my problem is solved in the sense that the sabine debugsymbols are found when using the --debugPath option, but it would be nice if I could also get rid of the other parse errors.

Also, does this give you enough information to know whether this is a bug or not? Is there anything else you want me to test or verify?

@milianw
Copy link
Member

milianw commented Oct 1, 2023

Can you please strace again with the latest appimage and paste an update situation, for when you only pass --sysroot?

Ten, can you try and see what happens when you clear your local /root/.debug/ path? Strace again and paste, as well as describe whether that changes any behavior.

Regarding the kernel symbol mapping file: you will have to specify that one manually via --kallsyms /proc/kallsyms, as it's not part of your sysroot.

@BartVandewoestyne
Copy link
Author

Sorry for the late reply... it's been kinda busy here at work ;-)

I don't know why, but trying to strace with the latest appimage doesn't seem to work. If I try

bvandewoestyne@BVANDEW-70TZQ73:~$ sudo strace -e file -f /home/bvandewoestyne/bin/hotspot-v1.4.1-191-gc795618-x86_64.AppImage --sysroot=`docker inspect -f '{{.GraphDriver.Data.MergedDir}}' docker_files_ins_200_113_chassis_1_inet_1_terminal_1_1` perf.data |& grep sabine

then my hotspot GUI doesn't start and I also don't see strace output in my terminal. It simply seems to 'hang'. Do you have any idea why? Probably has to do with the fact that its an AppImage...

With the hotspot version that I can install as a package on my Ubuntu Ubuntu 20.04.6 LTS, which is:

bvandewoestyne@BVANDEW-70TZQ73:~$ hotspot --version
hotspot 1.1.0

I get:

bvandewoestyne@BVANDEW-70TZQ73:~$ sudo strace -e file -f hotspot --sysroot=`docker inspect -f '{{.GraphDriver.Data.MergedDir}}' docker_files_ins_200_113_chassis_1_inet_1_terminal_1_1` perf.data |& grep sabine
[pid 608816] statx(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug//usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7ffd898256b0) = -1 ENOENT (No such file or directory)
[pid 608816] statx(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7ffd898256b0) = -1 ENOENT (No such file or directory)
[pid 608816] statx(AT_FDCWD, "/root/.debug//usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_ALL|0x1000, stx_attributes=0, stx_mode=S_IFDIR|0755, stx_size=4096, ...}) = 0
[pid 608816] statx(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_ALL|0x1000, stx_attributes=0, stx_mode=S_IFREG|0755, stx_size=9104232, ...}) = 0
[pid 608816] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = 5
[pid 608816] statx(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug//usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7ffd898256b0) = -1 ENOENT (No such file or directory)
[pid 608816] statx(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7ffd898256b0) = -1 ENOENT (No such file or directory)
[pid 608816] statx(AT_FDCWD, "/root/.debug//usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_ALL|0x1000, stx_attributes=0, stx_mode=S_IFDIR|0755, stx_size=4096, ...}) = 0
[pid 608816] statx(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_ALL|0x1000, stx_attributes=0, stx_mode=S_IFREG|0755, stx_size=9104232, ...}) = 0
[pid 608816] statx(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug//usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7ffd898256b0) = -1 ENOENT (No such file or directory)
[pid 608816] statx(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7ffd898256b0) = -1 ENOENT (No such file or directory)
[pid 608816] statx(AT_FDCWD, "/root/.debug//usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_ALL|0x1000, stx_attributes=0, stx_mode=S_IFDIR|0755, stx_size=4096, ...}) = 0
[pid 608816] statx(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_ALL|0x1000, stx_attributes=0, stx_mode=S_IFREG|0755, stx_size=9104232, ...}) = 0
[pid 608816] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = 8
[pid 608816] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/root/.debug/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/root/.debug/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/root/.debug/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/root/.debug/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = 9
[pid 608816] openat(AT_FDCWD, "/root/.debug/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/root/.debug/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/root/.debug/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/root/.debug/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/.debug/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/.debug/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = 9
[pid 608816] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOTDIR (Not a directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOTDIR (Not a directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOTDIR (Not a directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOTDIR (Not a directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] lstat("/root/.debug/usr/sbin/sabine", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 608816] lstat("/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 608816] lstat("/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", {st_mode=S_IFREG|0755, st_size=9104232, ...}) = 0
[pid 608816] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/root/.debug/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/root/.debug/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/root/.debug/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/root/.debug/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = 9
[pid 608816] openat(AT_FDCWD, "/root/.debug/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/root/.debug/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/root/.debug/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/root/.debug/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/.debug/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/.debug/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = 9
[pid 608816] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOTDIR (Not a directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOTDIR (Not a directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOTDIR (Not a directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOTDIR (Not a directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf.debug", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", O_RDONLY) = -1 ENOENT (No such file or directory)
[pid 608816] lstat("/root/.debug/usr/sbin/sabine", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 608816] lstat("/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 608816] lstat("/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", {st_mode=S_IFREG|0755, st_size=9104232, ...}) = 0

If I then do

bvandewoestyne@BVANDEW-70TZQ73:~$ sudo mv /root/.debug ~/debugbackup
bvandewoestyne@BVANDEW-70TZQ73:~$ sudo ls -al /root/.debug
ls: cannot access '/root/.debug': No such file or directory

and try again, I get:

bvandewoestyne@BVANDEW-70TZQ73:~$ sudo strace -e file -f hotspot --sysroot=`docker inspect -f '{{.GraphDriver.Data.MergedDir}}' docker_files_ins_200_113_chassis_1_inet_1_terminal_1_1` perf.data |& grep sabine
[pid 611688] statx(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug//usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7fffa301a360) = -1 ENOENT (No such file or directory)
[pid 611688] statx(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7fffa301a360) = -1 ENOENT (No such file or directory)
[pid 611688] statx(AT_FDCWD, "/root/.debug//usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7fffa301a360) = -1 ENOENT (No such file or directory)
[pid 611688] statx(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7fffa301a360) = -1 ENOENT (No such file or directory)
[pid 611688] statx(AT_FDCWD, "/home/bvandewoestyne/.debug//usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7fffa301a360) = -1 ENOENT (No such file or directory)
[pid 611688] statx(AT_FDCWD, "/home/bvandewoestyne/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7fffa301a360) = -1 ENOENT (No such file or directory)
[pid 611688] statx(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/sbin/sabine", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_ALL|0x1000, stx_attributes=0, stx_mode=S_IFREG|0755, stx_size=179067696, ...}) = 0
[pid 611688] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/sbin/sabine", O_RDONLY) = 5
[pid 611688] statx(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug//usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7fffa301a360) = -1 ENOENT (No such file or directory)
[pid 611688] statx(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7fffa301a360) = -1 ENOENT (No such file or directory)
[pid 611688] statx(AT_FDCWD, "/root/.debug//usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7fffa301a360) = -1 ENOENT (No such file or directory)
[pid 611688] statx(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7fffa301a360) = -1 ENOENT (No such file or directory)
[pid 611688] statx(AT_FDCWD, "/home/bvandewoestyne/.debug//usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7fffa301a360) = -1 ENOENT (No such file or directory)
[pid 611688] statx(AT_FDCWD, "/home/bvandewoestyne/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7fffa301a360) = -1 ENOENT (No such file or directory)
[pid 611688] statx(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/sbin/sabine", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_ALL|0x1000, stx_attributes=0, stx_mode=S_IFREG|0755, stx_size=179067696, ...}) = 0
[pid 611688] statx(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug//usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7fffa301a360) = -1 ENOENT (No such file or directory)
[pid 611688] statx(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/lib/debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7fffa301a360) = -1 ENOENT (No such file or directory)
[pid 611688] statx(AT_FDCWD, "/root/.debug//usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7fffa301a360) = -1 ENOENT (No such file or directory)
[pid 611688] statx(AT_FDCWD, "/root/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7fffa301a360) = -1 ENOENT (No such file or directory)
[pid 611688] statx(AT_FDCWD, "/home/bvandewoestyne/.debug//usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7fffa301a360) = -1 ENOENT (No such file or directory)
[pid 611688] statx(AT_FDCWD, "/home/bvandewoestyne/.debug/usr/sbin/sabine/b99b1e91087b5c7bbed0b2f71f819c5dbef11e9d/elf", AT_STATX_SYNC_AS_STAT, STATX_ALL, 0x7fffa301a360) = -1 ENOENT (No such file or directory)
[pid 611688] statx(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/sbin/sabine", AT_STATX_SYNC_AS_STAT, STATX_ALL, {stx_mask=STATX_ALL|0x1000, stx_attributes=0, stx_mode=S_IFREG|0755, stx_size=179067696, ...}) = 0
[pid 611688] openat(AT_FDCWD, "/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/sbin/sabine", O_RDONLY) = 7
[pid 611688] lstat("/var/lib/docker/overlay2/fa16200fc0196afd8dc742a7020a5279edc1dd15f9e83a98b57ef1c24aef59c1/merged/usr/sbin/sabine", {st_mode=S_IFREG|0755, st_size=179067696, ...}) = 0

In the second case, the sabine symbols were shown in the hotspot gui! So it looks like it indeed has to do with this /root/.debug directory... Not sure though whether this is a bug or a problem on my side (with the /root.debug directory).

Does this info help? Or do you want me to test other things?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@milianw @BartVandewoestyne