"bpftune is not supported" #37
Comments
|
I note that this situation is referenced in I also have |
|
Does Does it look in
|
|
it's looking for vmlinux BTF which is usually available in /sys/kernel/btf/vmlinux. BTF for kernel/modules is made available here via in-kernel mechanisms. does your container have a way to make /sys/kernel/btf available? |
|
The host outside the container doesn't have this either 😲
|
|
Is CONFIG_DEBUG_INFO_BTF specified? That's needed to get /sys/kernel/btf/vmlinux |
|
Sorry for the delay here - I've only just come back to this issue… I've looked at my kernel configuration, and to enable |
BPF Type Format (BTF) is needed for attaching higher-performance BPF programs such as "fentry/fexit" (kernel function entry/exit) and "tp_btf" (tracepoints). It's possible to force bpftune legacy mode via the "-L" option - this means you use slower kprobes instead of fentry, etc. I haven't tested it without BTF as most distros ship with BTF these days, but you could give that a try. I'm curious about which debug options are causing you pain - the aim is to make BTF tenable for as many systems as possible (including embedded), so any feedback on what is problematic is useful. If the problem is DWARF debug info, note that most distros strip it out of the vmlinux image and deliver it via a separate debuginfo package - BTF is generated via DWARF, but once generated it doesn't need it to hang around in the kernel image. This usually happens as part of the packaging process however, so if you're building kernels manually you might not get this benefit. I belatedly discovered "sudo make INSTALL_MOD_STRIP=1 modules_install install" which does the job for a hand-built kernel, maybe that might help. Or maybe the issue is something else? |
Ah, interesting, thanks - I'll give that a try! I guess this requires Do you anticipate that bpftune might itself be materially affected by using the slower mechanism, or are even slower kprobes more than fast enough for this tool's usage patterns?
The greatest problem I found is that somewhat memory-constrained systems (and we're talking 4GB, so not actually that small) would OOM when trying to link a kernel with debug enabled. Recently , the BPF options required
Yeah, I roll my own kernels and some of the hardware targets are systems that by today's standards would likely be considered 'embedded' (including hosts used as routers, various flavours of Raspberry Pi, etc.) I've recently updated my container-based build-system to use I'll admit that I don't fully understand whether enabling the kernel DEBUG options discussed here solely adds additional strippable sections to the resulting kernel binaries, or whether they also add additional code-paths which may impact performance - in case of the latter, is there any way to get libbpf/bpftune to use a (I ask these questions mostly in the hope that they might be useful for writing "An Idiots' Guide to Getting Started with bpftune" at some point in the future! 😅) |
|
(Argh, I typed a lengthy reply to this and GitHub lost it when the page refreshed 😩)
Ah, good to know - I'll try to test this. I guess
I've just done a recompile, and with the only change being setting The main issue, however, is that it takes significantly more memory to compile with the updated configuration - whereas the kernel without debug options can compile comfortably on a 4GB system (and possibly even a 1GB system!), I had to increase the memory allocated to my container-based build system to over 8GB (12GB worked, I didn't try to bisect the difference) in order for linking Therefore, if there were a means to use a |
|
thanks for collecting this data, it's great to know! the problem is that using the current scheme, DWARF is required to generate BTF, and the process of going from DWARF->BTF is very compute-intensive and takes a large amount of memory. I suspect the memory utilization peaks during the execution of "pahole" to generate BTF data. I'll mention your experience to the pahole maintainer; there may be ways to ensure compilation still works (albeit more slowly); 4GB should be enough to build the kernel. I'm interested in the increased sizes you observe in kernel/modules; a useful comparison might be to "objdump -h vmlinux" (i.e. the uncompressed kernel) in both cases to see if additional ELF sections are still present after stripping that are not there for the non-DWARF case. Returning to bpftune itself, I did some experiments with a kernel with no BTF, and "bpftune -S" correctly identifies that legacy mode should be used, but the tuners still fail to load. I'm looking for the root cause of that currently.. |
|
I've spent a fair bit of time getting bpftune to run without BTF, and I'm still hitting some walls. Even if we could get this to work, the key problem is we lose the benefit of Compile Once - Run Everywhere (CO-RE) that comes with BTF and that introduces risk that BPF programs that were built on kernel X will fail on kernel Y as the data structure offsets are different. I've pursued the approach of replacing use of BPF_CORE_READ() with BPF_PROBE_READ() - the latter of which is the non-CO-RE variant, but oddly I'm still seeing CO-RE relocations, and hence the program doesn't load without BTF (since CO-RE usage mandates BTF). So I suggest for now we
Will probably be a week or two till I can work on this again due to other commitments, but hopefully we can figure it out. |
|
I added a "no BTF" mode of operation to bpftune; it's merged into main now so could you pull, rebuild and see if it works for you? thanks! |
|
This is the |
|
A quick note on kernel builds: Without Once enabled, the output reads as follows: … are these |
|
we ran into multiple ID issues with pahole v1.24, but these should mostly have been resolved with pahole v1.25. Can you do a few things if you get a chance
we've got a few reports of issues like this but haven't been able to reproduce them yet unfortunately. thanks! |
(LLVM is at v14 due to kernel rust dependencies) |
|
Update on this: I still get lots of warning output very similar to the above from |
|
thanks for the update! I wasn't able to reproduce the issue with multiple ids using your config, but the cause of that in the past has been issues with the algorithm used to remove duplicate data structures (termed BTF de-duplication). if you get a chance, check if bpftune works on a BTF-less system; it does for me, but would be good to confirm that part of the problem is solved on your side too. thanks again! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I'm trying to run
bpftunefrom a--privilegedcontainer (rootful, via podman) on kernel 6.3.6 with the following configuration:… and after building from commit 9c99014 I get the following:
With:
The problem appears to be:
… but what can cause this?
The text was updated successfully, but these errors were encountered: