Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Compile modern_bpf error #1719

Open
1 of 2 tasks
yzewei opened this issue Mar 1, 2024 · 34 comments
Open
1 of 2 tasks

[BUG] Compile modern_bpf error #1719

yzewei opened this issue Mar 1, 2024 · 34 comments
Assignees
Labels
kind/bug Something isn't working
Milestone

Comments

@yzewei
Copy link
Contributor

yzewei commented Mar 1, 2024

Describe the bug
make drivers_test

/home/yzw/rd/libs/driver/modern_bpf/helpers/extract/extract_from_kernel.h:648:49: error: member reference base type 'struct percpu_counter[4]' is not a structure or union
  648 |                 BPF_CORE_READ_INTO(&swap_entries, mm, rss_stat.count[MM_SWAPENTS].counter);
      |                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~

The code files involved include :
modern_bpf/programs/tail_called/events/syscall_dispatched_events/execve.bpf.c modern_bpf/programs/tail_called/events/syscall_dispatched_events/execveat.bpf.c modern_bpf/helpers/extract/extract_from_kernel.h

How to reproduce it
First judge the kernel version, and then execute the corresponding operation. like this:

#if LINUX_VERSION_CODE >= KERNEL_VERSION(6, 3, 0)

instead of

if(bpf_core_field_exists(((struct kernel_cap_struct *)0)->cap)){...}

Expected behaviour

Screenshots

Environment

  • Falco version:
  • System info:
  • Cloud provider or hardware configuration:
  • OS:
  • Kernel:
  • Installation method:

Additional context

Tasks

  1. area/driver-kmod dco-signoff: yes do-not-merge/hold release-note-none size/XXL

Tasks

@yzewei yzewei added the kind/bug Something isn't working label Mar 1, 2024
@Andreagit97
Copy link
Member

Thank you for reporting, can you provide more info?

  • Which kernel version are you running?
  • Which architecture are you using?
  • Can you generate the vmlinux for your system with bpftool bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h ?

@yzewei
Copy link
Contributor Author

yzewei commented Mar 5, 2024

Thank you for reporting, can you provide more info?

  • Which kernel version are you running?
  • Which architecture are you using?
  • Can you generate the vmlinux for your system with bpftool bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h ?

Thank you for your attention!
The kernel version we are currently using is:

  • 6.8.0-rc5
  • The architecture is loongarch64, which is currently being adapted, including kmod, bpf, and modern-bpf modules
  • I can get vmlinux.h

@Andreagit97
Copy link
Member

Uhm ok looking at this compiler error

/home/yzw/rd/libs/driver/modern_bpf/helpers/extract/extract_from_kernel.h:648:49: error: member reference base type 'struct percpu_counter[4]' is not a structure or union
  648 |                 BPF_CORE_READ_INTO(&swap_entries, mm, rss_stat.count[MM_SWAPENTS].counter);
      |                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~

it seems that even if your kernel supports the new rss_stat format (so struct percpu_counter instead of struct mm_rss_stat) the struct mm_rss_stat is not removed in your kernel and for this reason, we enter in the wrong branch... we should enter in the else but according to the compiler error we are falling in the first one...

	if(bpf_core_type_exists(struct mm_rss_stat))
	{
		BPF_CORE_READ_INTO(&swap_entries, mm, rss_stat.count[MM_SWAPENTS].counter);
	}
	else
	{
		struct mm_struct___v6_2 *mm_v6_2 = (void *)mm;
		BPF_CORE_READ_INTO(&swap_entries, mm_v6_2, rss_stat[MM_SWAPENTS].count);
	}

If you run these 2 commands on your machine, they should both return true, can you check it?

bpftool btf dump file /sys/kernel/btf/vmlinux format c | grep -q "mm_rss_stat" && echo "true" || echo "false" 
bpftool btf dump file /sys/kernel/btf/vmlinux format c | grep -q "percpu_counter rss_stat" && echo "true" || echo "false" 

@yzewei
Copy link
Contributor Author

yzewei commented Mar 7, 2024

Uhm ok looking at this compiler error

/home/yzw/rd/libs/driver/modern_bpf/helpers/extract/extract_from_kernel.h:648:49: error: member reference base type 'struct percpu_counter[4]' is not a structure or union
  648 |                 BPF_CORE_READ_INTO(&swap_entries, mm, rss_stat.count[MM_SWAPENTS].counter);
      |                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~

it seems that even if your kernel supports the new rss_stat format (so struct percpu_counter instead of struct mm_rss_stat) the struct mm_rss_stat is not removed in your kernel and for this reason, we enter in the wrong branch... we should enter in the else but according to the compiler error we are falling in the first one...

	if(bpf_core_type_exists(struct mm_rss_stat))
	{
		BPF_CORE_READ_INTO(&swap_entries, mm, rss_stat.count[MM_SWAPENTS].counter);
	}
	else
	{
		struct mm_struct___v6_2 *mm_v6_2 = (void *)mm;
		BPF_CORE_READ_INTO(&swap_entries, mm_v6_2, rss_stat[MM_SWAPENTS].count);
	}

If you run these 2 commands on your machine, they should both return true, can you check it?

bpftool btf dump file /sys/kernel/btf/vmlinux format c | grep -q "mm_rss_stat" && echo "true" || echo "false" 
bpftool btf dump file /sys/kernel/btf/vmlinux format c | grep -q "percpu_counter rss_stat" && echo "true" || echo "false" 

@Andreagit97 Certainly!
According to the command, the result can be obtained as follows:

[yzw@fedora libs]$ bpftool btf dump file /sys/kernel/btf/vmlinux format c | grep -q "mm_rss_stat" && echo "true" || echo "false" 
bpftool btf dump file /sys/kernel/btf/vmlinux format c | grep -q "percpu_counter rss_stat" && echo "true" || echo "false" 
false
true

It's true that I didn't enter the else but entered the first branch. I'm trying to clear it. This is why...
In the vmlinux.h file there is really only struct percpu_counter rss_stat [4];
There was no mm_rss_stat

[yzw@fedora libs]$ find -name "vmlinux.h" | grep -rn "mm_rss_stat"
driver/modern_bpf/definitions/ppc64le/vmlinux.h:2125:struct mm_rss_stat {
driver/modern_bpf/definitions/ppc64le/vmlinux.h:2191:		struct mm_rss_stat rss_stat;
driver/modern_bpf/definitions/x86_64/vmlinux.h:1754:struct mm_rss_stat {
driver/modern_bpf/definitions/x86_64/vmlinux.h:1847:		struct mm_rss_stat rss_stat;
driver/modern_bpf/definitions/aarch64/vmlinux.h:2355:struct mm_rss_stat {
driver/modern_bpf/definitions/aarch64/vmlinux.h:2569:		struct mm_rss_stat rss_stat;
driver/modern_bpf/definitions/s390x/vmlinux.h:1668:struct mm_rss_stat {
driver/modern_bpf/definitions/s390x/vmlinux.h:1921:		struct mm_rss_stat rss_stat;
driver/modern_bpf/helpers/extract/extract_from_kernel.h:622:	 * `struct mm_rss_stat` doesn't exist anymore.
driver/modern_bpf/helpers/extract/extract_from_kernel.h:624:	if(bpf_core_type_exists(struct mm_rss_stat))

[yzw@fedora loongarch64]$ find -name "vmlinux.h" | grep -rn "rss_stat"
vmlinux.h:1558:		struct percpu_counter rss_stat[4];
vmlinux.h:99637:struct trace_event_raw_rss_stat {
vmlinux.h:99668:struct trace_event_data_offsets_rss_stat {};
vmlinux.h:99690:typedef void (*btf_trace_rss_stat)(void *, struct mm_struct *, int);

@yzewei
Copy link
Contributor Author

yzewei commented Mar 7, 2024

I found that the reason is that the path to percpu_counter was not imported at compile time
It exists in the compiled kernel folder, but when I install it using make headers_install it is not installed at the path/usr/include/linux.
And when I specify the path to percpu_counter as C_INCLUDE_PATH, there is a conflict at compiling zlib

[  1%] Performing build step for 'zlib'
In file included from ./zconf.h:450,
                 from ./zlib.h:34,
                 from test/example.c:8:
/usr/include/sys/types.h:42:18: 错误:conflicting types for ‘loff_t’; have ‘__loff_t’ {或称 ‘long int’}
   42 | typedef __loff_t loff_t;
      |                  ^~~~~~
In file included from /home/yzw/workspace/linux-chenhuacai/include/linux/limits.h:6,
                 from /usr/include/bits/local_lim.h:38,
                 from /usr/include/bits/posix1_lim.h:161,
                 from /usr/include/limits.h:195,
                 from /usr/lib/gcc/loongarch64-redhat-linux/13/include/limits.h:205,
                 from /usr/lib/gcc/loongarch64-redhat-linux/13/include/syslimits.h:7,
                 from /usr/lib/gcc/loongarch64-redhat-linux/13/include/limits.h:34,
                 from ./zconf.h:424:
/home/yzw/workspace/linux-chenhuacai/include/linux/types.h:52:33: 附注:previous declaration of ‘loff_t’ with type ‘loff_t’ {或称 ‘long long int’}
   52 | typedef __kernel_loff_t         loff_t;
      |                                 ^~~~~~
/usr/include/sys/types.h:59:17: 错误:conflicting types for ‘dev_t’; have ‘__dev_t’ {或称 ‘long unsigned int’}
   59 | typedef __dev_t dev_t;
      |                 ^~~~~
/home/yzw/workspace/linux-chenhuacai/include/linux/types.h:21:33: 附注:previous declaration of ‘dev_t’ with type ‘dev_t’ {或称 ‘unsigned int’}
   21 | typedef __kernel_dev_t          dev_t;
      |                                 ^~~~~
In file included from /usr/include/sys/types.h:130:
/usr/include/bits/types/timer_t.h:7:19: 错误:conflicting types for ‘timer_t’; have ‘__timer_t’ {或称 ‘void *’}
    7 | typedef __timer_t timer_t;
      |                   ^~~~~~~
/home/yzw/workspace/linux-chenhuacai/include/linux/types.h:31:33: 附注:previous declaration of ‘timer_t’ with type ‘timer_t’ {或称 ‘int’}
   31 | typedef __kernel_timer_t        timer_t;
      |                                 ^~~~~~~
In file included from /usr/include/sys/types.h:155:
/usr/include/bits/stdint-intn.h:27:19: 错误:conflicting types for ‘int64_t’; have ‘__int64_t’ {或称 ‘long int’}
   27 | typedef __int64_t int64_t;

What should I do about it?

@Andreagit97
Copy link
Member

Sorry, I didn't get it... if mm_rss_stat is not defined in your kernel, how is it possible that you enter the if statement?

I found that the reason is that the path to percpu_counter was not imported at compile time

I don't understand how this is related to the fact the the code finds a definition of mm_rss_stat 🤔
Could you provide the dump of your local vmlinux.h (bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h)? I don't understand who is defining this mm_rss_stat ...

@yzewei
Copy link
Contributor Author

yzewei commented Mar 19, 2024

@Andreagit97
Could this be due to the Loongarch architecture being different from other architectures? Is there any way I can debug bpf_core_type_exists?
Since I'm not very familiar with ebpf, I can't read the progress bar down here...

@Andreagit97
Copy link
Member

Andreagit97 commented Mar 20, 2024

Could this be due to the Loongarch architecture being different from other architectures? Is there any way I can debug bpf_core_type_exists?

Uhm i'm not sure this is an issue with bpf_core_type_exists, in the end, this is a simple libbpf macro that calls a compiler built-in

/*
 * Convenience macro to check that provided named type
 * (struct/union/enum/typedef) exists in a target kernel.
 * Returns:
 *    1, if such type is present in target kernel's BTF;
 *    0, if no matching type is found.
 */
#define bpf_core_type_exists(type)					    \
	__builtin_preserve_type_info(*(typeof(type) *)0, BPF_TYPE_EXISTS)

The best way to debug is to obtain the vmlinux.h of your kernel and see if this type is here or not.
I've seen in our previous conversation that you were able to type these commands

bpftool btf dump file /sys/kernel/btf/vmlinux format c | grep -q "mm_rss_stat" && echo "true" || echo "false" 
bpftool btf dump file /sys/kernel/btf/vmlinux format c | grep -q "percpu_counter rss_stat" && echo "true" || echo "false" 

Can you just run this command and provide us with the vmlinux.h?

bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h

@yzewei
Copy link
Contributor Author

yzewei commented Mar 20, 2024

The root cause of the problem is the problem of the clang built-in function under the loongarch architecture

@yzewei
Copy link
Contributor Author

yzewei commented Mar 20, 2024

Could this be due to the Loongarch architecture being different from other architectures? Is there any way I can debug bpf_core_type_exists?

Uhm i'm not sure this is an issue with bpf_core_type_exists, in the end, this is a simple macro that calls a compiler built-in

/*
 * Convenience macro to check that provided named type
 * (struct/union/enum/typedef) exists in a target kernel.
 * Returns:
 *    1, if such type is present in target kernel's BTF;
 *    0, if no matching type is found.
 */
#define bpf_core_type_exists(type)					    \
	__builtin_preserve_type_info(*(typeof(type) *)0, BPF_TYPE_EXISTS)

The best way to debug is to obtain the vmlinux.h of your kernel and see if this type is here or not. I've seen in our previous conversation that you were able to type these commands

bpftool btf dump file /sys/kernel/btf/vmlinux format c | grep -q "mm_rss_stat" && echo "true" || echo "false" 
bpftool btf dump file /sys/kernel/btf/vmlinux format c | grep -q "percpu_counter rss_stat" && echo "true" || echo "false" 

Can you just run this command and provide us with the vmlinux.h?

bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h

of course,https://github.com/libbpf/libbpf-bootstrap/blob/master/vmlinux/loongarch/vmlinux_602.h

@Andreagit97 Andreagit97 self-assigned this Mar 20, 2024
@Andreagit97 Andreagit97 added this to the TBD milestone Mar 20, 2024
@yzewei
Copy link
Contributor Author

yzewei commented Mar 20, 2024

@Andreagit97 If you would like my vmlinux.h from TCE-metal, I would be happy to provide it, please visit
https://github.com/Loongson-Cloud-Community/Loongson-Cloud-Community/blob/main/docs/%E5%85%B6%E4%BB%96%E6%96%87%E6%A1%A3/vmlinux.h

os:fedora
kernel-version:6.8.0-rc5
llvm-version:18
clang-version:18
BTF:on
CO-RE:on

Regarding this issue, I am consulting the relevant personnel in the llvm community responsible for the loongarch architecture.
If you have any suggestions, you can contact me at any time, thank you very much!
Although I don't know why the built-in function has such a problem, I think you can try loongarch's vmlinux.h replication problem under x86 architecture

@Andreagit97
Copy link
Member

uhm ok i see... not sure this is the exact vmlinux of you machine but yes, in this vmlinux you provided there is no the mm_rss_stat struct, so yes this is strange... I've seen that you are using an high clang version (18), have you ever tried with a lower version? something like clang-14/clang-15? Locally i use clang-14, maybe something is changed

@yzewei
Copy link
Contributor Author

yzewei commented Mar 20, 2024

uhm ok i see... not sure this is the exact vmlinux of you machine but yes, in this vmlinux you provided there is no the mm_rss_stat struct, so yes this is strange... I've seen that you are using an high clang version (18), have you ever tried with a lower version? something like clang-14/clang-15? Locally i use clang-14, maybe something is changed

Sir, I have provided the loongarch architecture vmlinux.h header file given by the libbpf official and the vmlinux.h generated by my machine respectively. Neither of these two header files from different sources mm_rss_stat structure

@yzewei
Copy link
Contributor Author

yzewei commented Mar 20, 2024

use clang-14 error:

error: unable to create target: 'No available targets are compatible with triple "bpf"'

@Andreagit97
Copy link
Member

ok, I read again the issue, and we are missing something... you are compiling this on your machine so what is happening is perfectly legit, the compiler is just saying that it cannot find a definition for struct mm_rss_stat in your vmlinux.h, so we need to add it between missing definitions, sorry for that, I was convinced for some reason that you were executing the probe against your local machine...

@yzewei
Copy link
Contributor Author

yzewei commented Mar 20, 2024

ok, I read again the issue, and we are missing something... you are compiling this on your machine so what is happening is perfectly legit, the compiler is just saying that it cannot find a definition for struct mm_rss_stat in your vmlinux.h, so we need to add it between missing definitions, sorry for that, I was convinced for some reason that you were executing the probe against your local machine...

Regarding this issue, if we wait for the built-in functions of clang to return to normal, it will take a long time. What are your suggestions?
Add arch detection to avoid this problem?
In #1654, we discussed the feasibility of switching to kernel version checking

@Andreagit97
Copy link
Member

Uhm let me explain better what is happening here. On your local machine, to compile the modern ebpf probe, you have probably added another vmlinux.h under this folder https://github.com/falcosecurity/libs/tree/0.15.0/driver/modern_bpf/definitions. In this vmlinux.h there is no definition for struct mm_rss_stat, while if you check in all the others vmlinux.h we have in this folder you will see that the struct mm_rss_stat is defined. This is because all the existing vmlinux.h are generated from a kernel version <= 6.2.

Now how can you solve this missing definition?
You need to add the definition by yourself in our struct_flavors.h file. So you need to add something like

#if defined(__TARGET_ARCH_loongarch)
struct mm_rss_stat {
	atomic_long_t count[4];
};
#endif

this should solve this compilation issue, but there could be others since your vmlinux.h is generated on a recent loongarch machine...

The best solution here would be to generate a vmlinux.h for loongarch in an older machine something with a kernel version <= 6.0.0, but not sure if you have a machine to do that

@yzewei
Copy link
Contributor Author

yzewei commented Mar 21, 2024

Uhm let me explain better what is happening here. On your local machine, to compile the modern ebpf probe, you have probably added another vmlinux.h under this folder https://github.com/falcosecurity/libs/tree/0.15.0/driver/modern_bpf/definitions. In this vmlinux.h there is no definition for struct mm_rss_stat, while if you check in all the others vmlinux.h we have in this folder you will see that the struct mm_rss_stat is defined. This is because all the existing vmlinux.h are generated from a kernel version <= 6.2.

Now how can you solve this missing definition? You need to add the definition by yourself in our struct_flavors.h file. So you need to add something like

#if defined(__TARGET_ARCH_loongarch)
struct mm_rss_stat {
	atomic_long_t count[4];
};
#endif

this should solve this compilation issue, but there could be others since your vmlinux.h is generated on a recent loongarch machine...

The best solution here would be to generate a vmlinux.h for loongarch in an older machine something with a kernel version <= 6.0.0, but not sure if you have a machine to do that

mm_rss_stat structure does not exist when kernel version > 6.2.0, this definition may cause problems with project compilation.
I will try to compile a lower version of the kernel

@yzewei
Copy link
Contributor Author

yzewei commented Mar 21, 2024

I wonder if we could change the criteria from mm_rss_stat to kernel version?
like this:
from

        if(bpf_core_type_exists(struct mm_rss_stat))
        {
                BPF_CORE_READ_INTO(&file_pages, mm, rss_stat.count[MM_FILEPAGES].counter);
                BPF_CORE_READ_INTO(&anon_pages, mm, rss_stat.count[MM_ANONPAGES].counter);
                BPF_CORE_READ_INTO(&shmem_pages, mm, rss_stat.count[MM_SHMEMPAGES].counter);
        }
        else
        {
                struct mm_struct___v6_2 *mm_v6_2 = (void *)mm;
                BPF_CORE_READ_INTO(&file_pages, mm_v6_2, rss_stat[MM_FILEPAGES].count);
                BPF_CORE_READ_INTO(&anon_pages, mm_v6_2, rss_stat[MM_ANONPAGES].count);
                BPF_CORE_READ_INTO(&shmem_pages, mm_v6_2, rss_stat[MM_SHMEMPAGES].count);
        }

to

#if LINUX_VERSION_CODE >= KERNEL_VERSION(6, 2, 0)
                BPF_CORE_READ_INTO(&file_pages, mm, rss_stat.count[MM_FILEPAGES].counter);
                BPF_CORE_READ_INTO(&anon_pages, mm, rss_stat.count[MM_ANONPAGES].counter);
                BPF_CORE_READ_INTO(&shmem_pages, mm, rss_stat.count[MM_SHMEMPAGES].counter);
#else
                struct mm_struct___v6_2 *mm_v6_2 = (void *)mm;
                BPF_CORE_READ_INTO(&file_pages, mm_v6_2, rss_stat[MM_FILEPAGES].count);
                BPF_CORE_READ_INTO(&anon_pages, mm_v6_2, rss_stat[MM_ANONPAGES].count);
                BPF_CORE_READ_INTO(&shmem_pages, mm_v6_2, rss_stat[MM_SHMEMPAGES].count);
#endif

@Andreagit97
Copy link
Member

I wonder if we could change the criteria from mm_rss_stat to kernel version?

Unfortunately no, because these changes could be backported in older kernel versions (so before 6.2.0), many distros do similar things... so the best way to detect these changes is to check the struct existence...

@yzewei
Copy link
Contributor Author

yzewei commented Mar 22, 2024

I wonder if we could change the criteria from mm_rss_stat to kernel version?

Unfortunately no, because these changes could be backported in older kernel versions (so before 6.2.0), many distros do similar things... so the best way to detect these changes is to check the struct existence...

After debugging, it was found that there was no exception when running the bpf_core_type_exists () function. For higher kernels that do not have mm_rss_stat structures, the result returned by the loongarch platform is 0.
But I still can't figure out why logic is true here

@yzewei
Copy link
Contributor Author

yzewei commented Mar 22, 2024

@Andreagit97
I found that the result of judging mm_rss_stat on loongarch is correct, just need to judge under the macro definition to be normal

        int result = bpf_core_type_exists(struct mm_rss_stat);
        /* In recent kernel versions (https://github.com/torvalds/linux/commit/f1a7941243c102a44e8847e3b94ff4ff3ec56f25)
         * `struct mm_rss_stat` doesn't exist anymore.
         */
#if result == 1
                BPF_CORE_READ_INTO(&file_pages, mm, rss_stat.count[MM_FILEPAGES].counter);
                BPF_CORE_READ_INTO(&anon_pages, mm, rss_stat.count[MM_ANONPAGES].counter);
                BPF_CORE_READ_INTO(&shmem_pages, mm, rss_stat.count[MM_SHMEMPAGES].counter);
#else
                struct mm_struct___v6_2 *mm_v6_2 = (void *)mm;
                BPF_CORE_READ_INTO(&file_pages, mm_v6_2, rss_stat[MM_FILEPAGES].count);
                BPF_CORE_READ_INTO(&anon_pages, mm_v6_2, rss_stat[MM_ANONPAGES].count);
                BPF_CORE_READ_INTO(&shmem_pages, mm_v6_2, rss_stat[MM_SHMEMPAGES].count);
#endif

@Andreagit97
Copy link
Member

Andreagit97 commented Mar 22, 2024

ei @yzewei i think you have 2 possible solutions as explained here #1719 (comment);

  1. introduce an #if defined(__TARGET_ARCH_loongarch) to add the missing structs definitions that the compiler needs.
  2. generate an old vmlinux.h for long arch (for old I mean a kernel version <= 6.2.0, probably <= 6.0 is even better since there are other structs that are changed) and put it in the repo.

If possible I would go for solution number 2, it would be easier for everyone since it should avoid ifdefs

@yzewei
Copy link
Contributor Author

yzewei commented Mar 25, 2024

ei @yzewei i think you have 2 possible solutions as explained here #1719 (comment);

  1. introduce an #if defined(__TARGET_ARCH_loongarch) to add the missing structs definitions that the compiler needs.
  2. generate an old vmlinux.h for long arch (for old I mean a kernel version <= 6.2.0, probably <= 6.0 is even better since there are other structs that are changed) and put it in the repo.

If possible I would go for solution number 2, it would be easier for everyone since it should avoid ifdefs

@Andreagit97 Compiling with vmlinux with a kernel smaller than 6.0 solved this problem, thank you for your guidance!!

@yzewei
Copy link
Contributor Author

yzewei commented Mar 25, 2024


@Andreagit97
Copy link
Member

@Andreagit97 version 6.1.0 of Linux is compatible with Loongarch starting from version 5.19. So I selected this version to compile and obtain the low-version vmlinux file.

Ok now that you have this vmlinux.h file, you should take it and come back to your initial setup with kernel version 6.8.0. Now you should add this file in a new folder loongarch here https://github.com/falcosecurity/libs/tree/master/driver/modern_bpf/definitions and you should try to compile again. This time many of the initial issues should be solved, at least the one reported in this issue. If not, please report the compilation error and the vmlinux.h you generated.

Unfortunately, this version may not support the BTF of Loongarch for the time being, which resulted in some issues during compilation.

Please note that you need to only take the vmlinux.h file from the old kernel version, the compilation should happen on your actual setup (kernel version 6.8.0). If the old machine doesn't support BTF you cannot compile the modern probe on it

@yzewei
Copy link
Contributor Author

yzewei commented Mar 25, 2024

@Andreagit97 version 6.1.0 of Linux is compatible with Loongarch starting from version 5.19. So I selected this version to compile and obtain the low-version vmlinux file.

Ok now that you have this vmlinux.h file, you should take it and come back to your initial setup with kernel version 6.8.0. Now you should add this file in a new folder loongarch here https://github.com/falcosecurity/libs/tree/master/driver/modern_bpf/definitions and you should try to compile again. This time many of the initial issues should be solved, at least the one reported in this issue. If not, please report the compilation error and the vmlinux.h you generated.

Unfortunately, this version may not support the BTF of Loongarch for the time being, which resulted in some issues during compilation.

Please note that you need to only take the vmlinux.h file from the old kernel version, the compilation should happen on your actual setup (kernel version 6.8.0). If the old machine doesn't support BTF you cannot compile the modern probe on it

Yes, I did it, I didn't encounter any other errors in the bpf part at present, but some data structures or data items do not exist due to kernel version problems. Under vmlinux.h, I added it in struct_flavors, let's see what problems will appear in the future

@yzewei
Copy link
Contributor Author

yzewei commented Mar 25, 2024

@Andreagit97 Here's another question:

/home/yzw/rd/libs/userspace/libpman/src/lifecycle.c: in function ‘pman_save_attached_progs’:
/home/yzw/rd/libs/userspace/libpman/src/lifecycle.c:44:76: error:‘struct <ambigous>’not member named sched_p_fork’
   44 |       g_state.attached_progs_fds[5] = bpf_program__fd(g_state.skel->progs.sched_p_fork);
      |                                                                          ^

@Andreagit97
Copy link
Member

ei @yzewei it seems like you have defined CAPTURE_SCHED_PROC_FORK somewhere for powerpc... it's difficult to say what you have done without seeing the code... can you open a PR with the changes so we can see it?

@yzewei
Copy link
Contributor Author

yzewei commented Mar 26, 2024

@Andreagit97 This problem is solved because system calls can be made under loongarch: sys_exit_execve
No need to add CAPTURE_SCHED_PROC_FORK

@yzewei
Copy link
Contributor Author

yzewei commented Mar 26, 2024

But there is still a problem when making bpf, the following is the problem situation

In file included from /home/yzw/rd/libs/driver/bpf/probe.c:27:
/home/yzw/rd/libs/driver/bpf/fillers.h:2369:48: warning: passing 'volatile long *' to parameter of type 'long *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]
 2369 |                 res = bpf_accumulate_argv_or_env(data, argv, &args_len);
      |                                                              ^~~~~~~~~
/home/yzw/rd/libs/driver/bpf/fillers.h:1923:19: note: passing argument to parameter 'args_len' here
 1923 |                                                       long *args_len)
      |                                                             ^
1 warning generated.
LLVM ERROR: Branch target out of insn range
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.	Program arguments: llc -march=bpf -filetype=obj -o /home/yzw/rd/libs/driver/bpf/probe.o /home/yzw/rd/libs/driver/bpf/probe.ll
 #0 0x00007fffe8a61acc llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/lib64/libLLVM-18.so+0xa61acc)
 #1 0x00007fffe8a5f9a0 llvm::sys::RunSignalHandlers() (/lib64/libLLVM-18.so+0xa5f9a0)
 #2 0x00007fffe8a5fd88 (/lib64/libLLVM-18.so+0xa5fd88)
 #3 0x00007ffffe79c89c (linux-vdso.so.1+0x89c)
 #4 0x00007fffeffaaf58 __pthread_kill_implementation.constprop.0 (/lib64/libc.so.6+0x82f58)
 #5 0x00007fffeff651e0 raise (/lib64/libc.so.6+0x3d1e0)
 #6 0x00007fffeff502f4 abort (/lib64/libc.so.6+0x282f4)
 #7 0x00007fffe89a4d74 llvm::report_fatal_error(llvm::StringRef, bool) (/lib64/libLLVM-18.so+0x9a4d74)
 #8 0x00007fffe89a4dec llvm::install_bad_alloc_error_handler(void (*)(void*, char const*, bool), void*) (/lib64/libLLVM-18.so+0x9a4dec)
 #9 0x00007fffeb34d19c (/lib64/libLLVM-18.so+0x334d19c)
#10 0x00007fffea214eec llvm::MCAssembler::layout(llvm::MCAsmLayout&) (/lib64/libLLVM-18.so+0x2214eec)
#11 0x00007fffea215060 llvm::MCAssembler::Finish() (/lib64/libLLVM-18.so+0x2215060)
#12 0x00007fffe93ba8c4 llvm::AsmPrinter::doFinalization(llvm::Module&) (/lib64/libLLVM-18.so+0x13ba8c4)
#13 0x00007fffe8baa138 llvm::FPPassManager::doFinalization(llvm::Module&) (/lib64/libLLVM-18.so+0xbaa138)
#14 0x00007fffe8bb53e4 llvm::legacy::PassManagerImpl::run(llvm::Module&) (/lib64/libLLVM-18.so+0xbb53e4)
#15 0x00005555587cff38 (/usr/bin/llc+0x17f38)
#16 0x00005555587c7bb4 main (/usr/bin/llc+0xfbb4)
#17 0x00007fffeff5082c __libc_start_call_main (/lib64/libc.so.6+0x2882c)
#18 0x00007fffeff50918 __libc_start_main@@GLIBC_2.36 (/lib64/libc.so.6+0x28918)
#19 0x00005555587c8140 _start (/usr/bin/llc+0x10140)
make[7]: *** [/home/yzw/rd/libs/driver/bpf/Makefile:56: /home/yzw/rd/libs/driver/bpf/probe.o]  Segmentation fault (core dumped)
make[6]: *** [/home/yzw/workspace/linux-chenhuacai/Makefile:1921:/home/yzw/rd/libs/driver/bpf] error 2
make[5]: *** [Makefile:240:__sub-make] error 2
make[4]: *** [Makefile:39:all] error 2
make[3]: *** [driver/bpf/CMakeFiles/bpf.dir/build.make:70:driver/bpf/CMakeFiles/bpf] error 2
make[2]: *** [CMakeFiles/Makefile2:859:driver/bpf/CMakeFiles/bpf.dir/all] error 2
make[1]: *** [CMakeFiles/Makefile2:866:driver/bpf/CMakeFiles/bpf.dir/rule] error 2
make: *** [Makefile:312:bpf] error 2

@yzewei
Copy link
Contributor Author

yzewei commented Mar 26, 2024

ei @yzewei it seems like you have defined CAPTURE_SCHED_PROC_FORK somewhere for powerpc... it's difficult to say what you have done without seeing the code... can you open a PR with the changes so we can see it?

At present, the code needs to be sorted out and reviewed internally, so it will take some time to update the relevant code on the previous PR. Please understand, thank you very much!

@yzewei
Copy link
Contributor Author

yzewei commented Apr 18, 2024

@Andreagit97 The problem has been identified, because the branch instructions compiled by llc need to encode offsets greater than 2^16 instructions, which is beyond the limit of cpuv3 and requires the use of cpuv4, and then depends on the higher version of the kernel. Other methods are being communicated so that the lower version kernel can use the bpf module of libs

@Andreagit97
Copy link
Member

oh, great catch! thank you for the update!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants