Request for information: How to proper cleanup on eBPF program termination/panic.

[lyuts]

Hi Team,

Thank you for a very handy library. I'm new to aya and eBPF in general. I'm trying to figure out whether my observations are expected, or I am missing some chunk of code that would properly cleanup after program termination. I created a project from the aya template and implemented an xdp program following the guide in the aya book. The program was executed several times, and it either got terminated by me, or panic'd. When I run bpftool prog I keep seeing entries for all of the times the program was run.

user@btfbpf:~/ws/aya-xdp$ sudo bpftool prog                       
...
20: xdp  name aya_xdp  tag b9a56507e863fa2f  gpl                  
        loaded_at 2023-11-17T19:39:05-0800  uid 0                 
        xlated 8936B  jited 4791B  memlock 12288B  map_ids 5,6,4  
26: xdp  name aya_xdp  tag fbaaa6cb56a49786  gpl                  
        loaded_at 2023-11-17T19:39:59-0800  uid 0                 
        xlated 1152B  jited 642B  memlock 4096B  map_ids 14,13,11 
32: xdp  name aya_xdp  tag 27c4a87cc5c8a088  gpl                  
        loaded_at 2023-11-17T19:40:52-0800  uid 0                 
        xlated 4872B  jited 2646B  memlock 8192B  map_ids 18,19,21
38: xdp  name aya_xdp  tag 74ee03a1ff3b8f67  gpl                  
        loaded_at 2023-11-17T19:43:51-0800  uid 0                 
        xlated 5928B  jited 3222B  memlock 8192B  map_ids 25,26,28
44: xdp  name aya_xdp  tag 1261cc8ab05fd22b  gpl                  
        loaded_at 2023-11-17T19:48:06-0800  uid 0                 
        xlated 7272B  jited 3914B  memlock 8192B  map_ids 32,34,35
50: xdp  name aya_xdp  tag 2bc91840faa3c0e7  gpl                  
        loaded_at 2023-11-17T19:48:29-0800  uid 0                 
        xlated 7288B  jited 3923B  memlock 8192B  map_ids 40,42,41

Is this expected? My impression is that this command would be showing only currently active programs. If the above is a sign of not properly cleaning up program's resources, could you point me to how cleanup is generally done with aya?

Any bit of information is appreciated. Thank you.

[alessandrod]

What kernel are you running on? Newer kernels (>= 5.9) will unload xdp programs when the creating process terminates, older kernels won't.

[lyuts]

I'm on 6.1

$ uname -a
Linux btfbpf 6.1.0-13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.55-1 (2023-09-29) x86_64 GNU/Linux

[alessandrod]

Odd, can you run your userspace program under strace -e bpf and paste the output?

[lyuts]

user@btfbpf:~/ws/aya-xdp$ sudo !!
sudo strace -e bpf ./target/debug/aya-xdp
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0\20\0\0\0\20\0\0\0\5\0\0\0\1\0\0\0\0\0\0\1"..., btf_log_buf=NULL, btf_size=45, btf_log_size=0, btf_log_level=0}, 144) = 9
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\08\0\0\08\0\0\0\r\0\0\0\1\0\0\0\0\0\0\1"..., btf_log_buf=NULL, btf_size=93, btf_log_size=0, btf_log_level=0}, 144) = 9
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\08\0\0\08\0\0\0\r\0\0\0\1\0\0\0\0\0\0\1"..., btf_log_buf=NULL, btf_size=93, btf_log_size=0, btf_log_level=0}, 144) = 9
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\08\0\0\08\0\0\0\17\0\0\0\1\0\0\0\0\0\0\1"..., btf_log_buf=NULL, btf_size=95, btf_log_size=0, btf_log_level=0}, 144) = 9
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0\f\0\0\0\f\0\0\0\7\0\0\0\1\0\0\0\0\0\0\20"..., btf_log_buf=NULL, btf_size=43, btf_log_size=0, btf_log_level=0}, 144) = 9
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0000\0\0\0000\0\0\0\22\0\0\0\1\0\0\0\0\0\0\1"..., btf_log_buf=NULL, btf_size=90, btf_log_size=0, btf_log_level=0}, 144) = 9
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0(\0\0\0(\0\0\0\5\0\0\0\0\0\0\0\0\0\0\1"..., btf_log_buf=NULL, btf_size=69, btf_log_size=0, btf_log_level=0}, 144) = 9
bpf(BPF_BTF_LOAD, {btf="\237\353\1\0\30\0\0\0\0\0\0\0\30\0\0\0\30\0\0\0\n\0\0\0\1\0\0\0\1\0\0\223"..., btf_log_buf=NULL, btf_size=58, btf_log_size=0, btf_log_level=0}, 144) = 9
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=2, insns=0x559e5fbdbff0, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="aya_name_check", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0, fd_array=NULL}, 144) = 9
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_TRACEPOINT, insn_cnt=6, insns=0x559e5fbcd7f0, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0, fd_array=NULL}, 144) = 9
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_TRACEPOINT, insn_cnt=2, insns=0x559e5fbcbf50, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0, fd_array=NULL}, 144) = 9
bpf(BPF_LINK_CREATE, {link_create={prog_fd=9, target_fd=-1, attach_type=BPF_PERF_EVENT, flags=0, perf_event={bpf_cookie=0}}}, 144) = -1 EBADF (Bad file descriptor)
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4, value_size=32, max_entries=1, map_flags=0, inner_map_fd=0, map_name="aya_global", map_ifindex=0, btf_fd=0, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0, map_extra=0}, 144) = 9
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_SOCKET_FILTER, insn_cnt=5, insns=0x559e5fbd2ae0, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0, fd_array=NULL}, 144) = 10
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_KPROBE, insn_cnt=2, insns=0x559e5fbcbf50, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(0, 0, 0), prog_flags=0, prog_name="", prog_ifindex=0, expected_attach_type=BPF_CGROUP_INET_INGRESS, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0, fd_array=NULL}, 144) = 9
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_CPUMAP, key_size=4, value_size=8, max_entries=1, map_flags=0, inner_map_fd=0, map_name="", map_ifindex=0, btf_fd=0, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0, map_extra=0}, 144) = 9
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_DEVMAP, key_size=4, value_size=8, max_entries=1, map_flags=0, inner_map_fd=0, map_name="", map_ifindex=0, btf_fd=0, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0, map_extra=0}, 144) = 9
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERCPU_ARRAY, key_size=4, value_size=8192, max_entries=1, map_flags=0, inner_map_fd=0, map_name="AYA_LOG_BUF", map_ifindex=0, btf_fd=0, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0, map_extra=0}, 144) = 9
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_PERF_EVENT_ARRAY, key_size=4, value_size=4, max_entries=4, map_flags=0, inner_map_fd=0, map_name="AYA_LOGS", map_ifindex=0, btf_fd=0, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0, map_extra=0}, 144) = 10
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_HASH, key_size=4, value_size=4, max_entries=1024, map_flags=0, inner_map_fd=0, map_name="BLOCKLIST", map_ifindex=0, btf_fd=0, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0, map_extra=0}, 144) = 11
bpf(BPF_MAP_CREATE, {map_type=BPF_MAP_TYPE_ARRAY, key_size=4, value_size=55, max_entries=1, map_flags=BPF_F_RDONLY_PROG, inner_map_fd=0, map_name=".rodata", map_ifindex=0, btf_fd=0, btf_key_type_id=0, btf_value_type_id=0, btf_vmlinux_value_type_id=0, map_extra=0}, 144) = 12
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=12, key=0x559e5e5f90dc, value=0x559e5fbcb2c0, flags=BPF_ANY}, 144) = 0
bpf(BPF_MAP_FREEZE, {map_fd=12}, 144)   = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=10, key=0x7ffc75c404ec, value=0x7ffc75c405c4, flags=BPF_ANY}, 144) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=10, key=0x7ffc75c404ec, value=0x7ffc75c405c4, flags=BPF_ANY}, 144) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=10, key=0x7ffc75c404ec, value=0x7ffc75c405c4, flags=BPF_ANY}, 144) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=10, key=0x7ffc75c404ec, value=0x7ffc75c405c4, flags=BPF_ANY}, 144) = 0
bpf(BPF_PROG_LOAD, {prog_type=BPF_PROG_TYPE_XDP, insn_cnt=911, insns=0x559e5fc231f0, license="GPL", log_level=0, log_size=0, log_buf=NULL, kern_version=KERNEL_VERSION(6, 1, 55), prog_flags=0, prog_name="aya_xdp", prog_ifindex=0, expected_attach_type=BPF_XDP, prog_btf_fd=0, func_info_rec_size=0, func_info=NULL, func_info_cnt=0, line_info_rec_size=0, line_info=NULL, line_info_cnt=0, attach_btf_id=0, attach_prog_fd=0, fd_array=NULL}, 144) = 17
bpf(BPF_LINK_CREATE, {link_create={prog_fd=17, target_fd=2, attach_type=BPF_XDP, flags=0}}, 144) = 18
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=11, key=0x7ffc75c410b0, value=0x7ffc75c410b4, flags=BPF_ANY}, 144) = 0
bpf(BPF_MAP_UPDATE_ELEM, {map_fd=11, key=0x7ffc75c410b0, value=0x7ffc75c410b4, flags=BPF_ANY}, 144) = 0
^Cstrace: Process 7383 detached

[alessandrod]

It is using BPF_LINK_CREATE as expected, so it should unlink on process termination. Or that was the case in 5.x at least, I'll have to check if behavior was changed. We have lifecycle integration tests here https://github.com/aya-rs/aya/blob/main/test/integration-test/src/tests/load.rs but we don't test killing a process, we probably should

[lyuts]

Thank you for the pointers. Let me get familiar with aya tests as well as kernel behavior.

[lyuts]

Quick follow up question, re: BPF_LINK_CREATE, is this the step after which a file is created in /sys/fs/bpf/ ? I have the program running, but that directory stays always empty:

user@btfbpf:~/ws/aya-xdp$ sudo find /sys/fs/bpf -type f
user@btfbpf:~/ws/aya-xdp$ 

(iirc correctly from other articles, guides, some files should exists there during the program execution.

Update: I think I'm mixing this with 'pinning'. If so, please ignore this.

[alessandrod]

Correct, it’s pinning that creates files in bpffs

[lyuts]

Tried running integration tests, with fresh main got 3 test cases fail. Test cases are for relocation, so this may be a different issue.

failures:

---- tests::btf_relocations::relocation_tests::_enum_signed_32_true_some_kernelversion_new_6_0_0_https_git-
thread 'tests::btf_relocations::relocation_tests::_enum_signed_32_true_some_kernelversion_new_6_0_0_https_:
assertion `left == right` failed
  left: 2219066437
 right: 18446744071633650757
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- tests::btf_relocations::relocation_tests::_enum_signed_64_true_some_kernelversion_new_6_0_0_https_git-
thread 'tests::btf_relocations::relocation_tests::_enum_signed_64_true_some_kernelversion_new_6_0_0_https_:
assertion `left == right` failed
  left: 572662307
 right: 17524406869737742883

---- tests::btf_relocations::relocation_tests::_enum_unsigned_64_true_some_kernelversion_new_6_0_0_https_g-
thread 'tests::btf_relocations::relocation_tests::_enum_unsigned_64_true_some_kernelversion_new_6_0_0_http:
assertion `left == right` failed
  left: 3722304989
 right: 14757395259253972445


failures:
    tests::btf_relocations::relocation_tests::_enum_signed_32_true_some_kernelversion_new_6_0_0_https_giths
    tests::btf_relocations::relocation_tests::_enum_signed_64_true_some_kernelversion_new_6_0_0_https_giths
    tests::btf_relocations::relocation_tests::_enum_unsigned_64_true_some_kernelversion_new_6_0_0_https_gis

[alessandrod]

what kernel are you running on and what command did you run to run the tests?

[lyuts]

I'm on 6.1, and ran tests as:

cargo xtask integration-test vm /boot/vmlinuz-6.1.0-12-amd64