new(pkg/driver): try to fetch kernel headers leveraging driverkit library when building drivers #476

FedeDP · 2024-03-13T08:08:06Z

What type of PR is this?

/kind feature

Any specific area of the project related to this PR?

/area library

What this PR does / why we need it:

This PR introduces a mechanism to leverage driverkit library to automatically download kernel headers for driverkit supported distros, when building drivers.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

pkg/driver/distro/distro.go

FedeDP · 2024-03-14T08:04:43Z

Output for a bpf probe build against drivers 6.0.1+driver on my archlinux machine using downloaded headers:

2024-03-14 09:00:59 INFO  Running falcoctl driver install
                      ├ driver version: 6.0.1+driver
                      ├ driver type: ebpf
                      ├ driver name: falco
                      ├ compile: true
                      ├ download: true
                      ├ arch: x86_64
                      ├ kernel release: 6.8.0-arch1-1
                      └ kernel version: #1 SMP PREEMPT_DYNAMIC Mon, 11 Mar 2024 03:12:55 +0000
2024-03-14 09:00:59 INFO  Found distro target: arch
2024-03-14 09:00:59 INFO  Removing eBPF probe symlink path: /root/.falco/falco-bpf.o
2024-03-14 09:00:59 INFO  Trying to download a driver. url: https://download.falco.org/driver/6.0.1%2Bdriver/x86_64/falco_arch_6.8.0-arch1-1_1.o
2024-03-14 09:01:00 WARN  Non-200 response from url. code: 404
2024-03-14 09:01:00 WARN  unable to find a prebuilt driver
2024-03-14 09:01:07 INFO  Mounting debugfs for bpf driver.
make: ingresso nella directory «/usr/src/falco-6.0.1+driver/bpf»
make -C /tmp/kernel M=$PWD
make[1]: ingresso nella directory «/tmp/kernel»
clang -I./arch/x86/include -I./arch/x86/include/generated  -I./include -I./arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/compiler-version.h -include ./include/linux/kconfig.h \
        -D__KERNEL__ -fmacro-prefix-map=./= \
         \
         \
        -D__KERNEL__ \
        -D__BPF_TRACING__ \
        -Wno-gnu-variable-sized-type-not-at-end \
        -Wno-address-of-packed-member \
        -fno-jump-tables \
        -fno-stack-protector \
        -Wno-tautological-compare \
        -Wno-unknown-attributes \
        -O2 -g -emit-llvm -c /usr/src/falco-6.0.1+driver/bpf/probe.c -o /usr/src/falco-6.0.1+driver/bpf/probe.ll
In file included from /usr/src/falco-6.0.1+driver/bpf/probe.c:16:
In file included from ./include/linux/sched.h:12:
In file included from ./arch/x86/include/asm/current.h:10:
In file included from ./include/linux/cache.h:6:
In file included from ./arch/x86/include/asm/cache.h:5:
In file included from ./include/linux/linkage.h:8:
In file included from ./arch/x86/include/asm/linkage.h:6:
./arch/x86/include/asm/ibt.h:77:8: warning: 'nocf_check' attribute ignored; use -fcf-protection to enable the attribute [-Wignored-attributes]
   77 | extern __noendbr u64 ibt_save(bool disable);
      |        ^
./arch/x86/include/asm/ibt.h:32:34: note: expanded from macro '__noendbr'
   32 | #define __noendbr       __attribute__((nocf_check))
      |                                        ^
./arch/x86/include/asm/ibt.h:78:8: warning: 'nocf_check' attribute ignored; use -fcf-protection to enable the attribute [-Wignored-attributes]
   78 | extern __noendbr void ibt_restore(u64 save);
      |        ^
./arch/x86/include/asm/ibt.h:32:34: note: expanded from macro '__noendbr'
   32 | #define __noendbr       __attribute__((nocf_check))
      |                                        ^
In file included from /usr/src/falco-6.0.1+driver/bpf/probe.c:26:
/usr/src/falco-6.0.1+driver/bpf/fillers.h:2851:22: error: no member named 'i_mtime' in 'struct inode'; did you mean '__i_mtime'?
 2851 |         time = _READ(inode->i_mtime);
      |                             ^~~~~~~
      |                             __i_mtime
/usr/src/falco-6.0.1+driver/bpf/plumbing_helpers.h:23:28: note: expanded from macro '_READ'
   23 | #define _READ(P) ({ typeof(P) _val;                                     \
      |                            ^
./include/linux/fs.h:677:20: note: '__i_mtime' declared here
  677 |         struct timespec64       __i_mtime;
      |                                 ^
In file included from /usr/src/falco-6.0.1+driver/bpf/probe.c:26:
/usr/src/falco-6.0.1+driver/bpf/fillers.h:2851:22: error: no member named 'i_mtime' in 'struct inode'; did you mean '__i_mtime'?
 2851 |         time = _READ(inode->i_mtime);
      |                             ^~~~~~~
      |                             __i_mtime
/usr/src/falco-6.0.1+driver/bpf/plumbing_helpers.h:24:51: note: expanded from macro '_READ'
   24 |                     bpf_probe_read_kernel(&_val, sizeof(_val), &P);     \
      |                                                                 ^
./include/linux/fs.h:677:20: note: '__i_mtime' declared here
  677 |         struct timespec64       __i_mtime;
      |                                 ^
2 warnings and 2 errors generated.
make[3]: *** [/usr/src/falco-6.0.1+driver/bpf/Makefile:54: /usr/src/falco-6.0.1+driver/bpf/probe.o] Error 1
make[2]: *** [/tmp/kernel/Makefile:1921: /usr/src/falco-6.0.1+driver/bpf] Error 2
make[1]: *** [Makefile:240: __sub-make] Error 2
make[1]: uscita dalla directory «/tmp/kernel»
make: *** [Makefile:38: all] Error 2
make: uscita dalla directory «/usr/src/falco-6.0.1+driver/bpf»
2024-03-14 09:01:08 ERROR failed: exit status 2

As you can see:

we use /tmp/kernel as src directory during the build
the build fails because driver 6.0.1+driver did not support kernel 6.8
/tmp/kernel and /tmp/kernel-download folders are cleaned up upon leaving

FedeDP · 2024-03-14T08:23:13Z

Oras bump was moved to #478.
Since driverkit master uses new oras, the bump is needed.

FedeDP · 2024-03-14T09:08:59Z

/milestone v0.8.0

FedeDP · 2024-03-14T14:11:36Z

Output for a kmod build (with dkms):

...
Kernel config /tmp/kernel/.config not found, modules won't be signed
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"
Creating symlink /var/lib/dkms/falco/6.0.1+driver/source -> /usr/src/falco-6.0.1+driver

Building module:
make -C /tmp/kernel M=/var/lib/dkms/falco/6.0.1+driver/build clean
make: ingresso nella directory «/tmp/kernel»
  CLEAN   /var/lib/dkms/falco/6.0.1+driver/build/Module.symvers
make: uscita dalla directory «/tmp/kernel»

{ '/tmp/falco-dkms-make'; } >> /var/lib/dkms/falco/6.0.1+driver/build/make.log 2>&1
(bad exit status: 2)
Error! Bad return status for module build on kernel: 6.8.0-arch1-1 (x86_64)
Consult /var/lib/dkms/falco/6.0.1+driver/build/make.log for more information.
2024-03-14 15:07:07 WARN  Running dkms build failed. Dumping dkms log. file: /var/lib/dkms/falco/6.0.1+driver/build/make.log
...

…ing driverkit library when building drivers. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

Also, bumped driverkit to falcosecurity/driverkit#324 HEAD. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

…", ...)`. Moreover, bumped driverkit to latest HEAD of PR324. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

FedeDP · 2024-03-25T14:53:17Z

Bumped to driverkit v0.18.0 that contains needed change.
Dropping wip now.

Moreover, added a couple of debug logs when automatically fetching headers. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

FedeDP · 2024-03-25T15:11:33Z

cmd/driver/install/install.go

-			"Detected an unsupported target system, please get in touch with the Falco community. Trying to compile anyway.")
-	} else {
-		return "", fmt.Errorf("detected an unsupported target system, please get in touch with the Falco community")
+	if o.Distro.String() == driverdistro.UndeterminedDistro {


We need to make the check only when distro is undetermined. This is a fix on top of current main (bug introduced in #484)

FedeDP · 2024-03-25T15:13:15Z

pkg/driver/distro/distro.go

@@ -128,6 +128,26 @@ func getOSReleaseDistro(kr *kernelrelease.KernelRelease) (Distro, error) {
 	return distro, nil
 }

+//nolint:gocritic // the method shall not be able to modify kr
+func loadKernelHeadersFromDk(distro string, kr kernelrelease.KernelRelease) (string, func(), error) {


Biggest change: try to load kernel headers from Driverkit, in a simple way:

instantiate driverkit builder for the current target distro

fetch the KernelDownloadScript from the builder, that is the driverkit script capable of downloading and extracting kernel headers for the distro

call the script; the KernelDownloadScript will output to stdout a single line containing the folder where the kernel headers were extracted

return a cleanup function to later remove the downloaded headers, and the headers path

FedeDP · 2024-03-25T15:13:30Z

pkg/driver/distro/distro.go

@@ -179,6 +199,25 @@ func Build(ctx context.Context,
 	if err != nil {
 		return "", err
 	}
+
+	if env == nil {


Avoid a segfault when later accessing the map.

FedeDP · 2024-03-25T15:15:21Z

pkg/driver/distro/distro.go

+
+	// If customizeBuild did not set any KERNELDIR env variable,
+	// try to load kernel headers urls from driverkit.
+	if _, found := env[drivertype.KernelDirEnv]; !found {


If no customizeBuild added the KERNELDIR env var to the env map, we try to autoamtically download and extract kernel headers leveraging driverkit; if we are able to do so, we emplace the new KERNELDIR env var pointing to the extraction path for the kernel headers.

FedeDP · 2024-03-25T15:15:47Z

pkg/driver/type/kmod.go

@@ -61,26 +53,26 @@ func (k *kmod) String() string {
 // Then, using dkms, it tries to fetch all
 // dkms-installed versions of the module to clean them up.
 func (k *kmod) Cleanup(printer *output.Printer, driverName string) error {
-	_, err := exec.Command("bash", "-c", "hash lsmod").Output()
+	lsmod, err := exec.LookPath("lsmod")


Switch to use exec.LookPath instead of the ugly bash magic.

FedeDP · 2024-03-25T15:16:19Z

pkg/driver/type/kmod.go

-			driverName, driverVersion, kr.String())
+		var dkmsCmdArgs string
+		if kernelDir, found := env[KernelDirEnv]; found {
+			dkmsCmdArgs = fmt.Sprintf(`dkms install --kernelsourcedir %q --directive="MAKE='/tmp/falco-dkms-make'" -m %q -v %q -k %q --verbose`,


If KERNELDIR env var was passed, forward it to dkms install too!

Note: ebpf did not need any change since it was already passing all the env to the make command.

poiana · 2024-03-28T08:13:39Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: FedeDP, leogr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [FedeDP,leogr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

poiana added kind/feature do-not-merge/work-in-progress dco-signoff: yes area/library labels Mar 13, 2024

poiana requested review from alacuku and loresuso March 13, 2024 08:08

poiana added approved size/L labels Mar 13, 2024

FedeDP commented Mar 13, 2024

View reviewed changes

pkg/driver/distro/distro.go Outdated Show resolved Hide resolved

FedeDP force-pushed the new/download_headers_from_dk branch from 5840e09 to aba70f8 Compare March 13, 2024 13:45

FedeDP mentioned this pull request Mar 14, 2024

new(pkg,cmd): refactored builder script logic. falcosecurity/driverkit#324

Merged

FedeDP force-pushed the new/download_headers_from_dk branch from c1ddf1f to 38b5d60 Compare March 14, 2024 08:13

FedeDP force-pushed the new/download_headers_from_dk branch from 38b5d60 to 62c3f83 Compare March 14, 2024 09:07

poiana added this to the v0.8.0 milestone Mar 14, 2024

FedeDP force-pushed the new/download_headers_from_dk branch from 04b52d5 to e6b4dfd Compare March 14, 2024 14:10

FedeDP mentioned this pull request Mar 25, 2024

new(pkg): use driverkit local build processor instead of implementing drivers build #356

Merged

4 tasks

FedeDP added 4 commits March 25, 2024 15:47

new(pkg/driver): initial work towards kernel headers download leverag…

abb4fe3

…ing driverkit library when building drivers. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

new(pkg): implemented kernel headers automatic download + extraction.

318d196

Also, bumped driverkit to falcosecurity/driverkit#324 HEAD. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

chore(pkg/driver): use exec.LookPath instead of `exec.Command("hash…

8f0a5f7

…", ...)`. Moreover, bumped driverkit to latest HEAD of PR324. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

chore: bumped driverkit to v0.18.0.

7e8faf5

Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

FedeDP force-pushed the new/download_headers_from_dk branch from e6b4dfd to 7e8faf5 Compare March 25, 2024 14:50

FedeDP changed the title ~~wip: new(pkg/driver): try to fetch kernel headers leveraging driverkit library when building drivers~~ new(pkg/driver): try to fetch kernel headers leveraging driverkit library when building drivers Mar 25, 2024

poiana removed the do-not-merge/work-in-progress label Mar 25, 2024

fix(cmd/driver,pkg/driver): fixed a check in install.go.

59afba7

Moreover, added a couple of debug logs when automatically fetching headers. Signed-off-by: Federico Di Pierro <nierro92@gmail.com>

FedeDP commented Mar 25, 2024

View reviewed changes

leogr approved these changes Mar 28, 2024

View reviewed changes

poiana assigned leogr Mar 28, 2024

poiana added the lgtm label Mar 28, 2024

poiana merged commit 064c201 into main Mar 28, 2024
17 checks passed

poiana deleted the new/download_headers_from_dk branch March 28, 2024 08:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

new(pkg/driver): try to fetch kernel headers leveraging driverkit library when building drivers #476

new(pkg/driver): try to fetch kernel headers leveraging driverkit library when building drivers #476

FedeDP commented Mar 13, 2024

FedeDP commented Mar 14, 2024

FedeDP commented Mar 14, 2024

FedeDP commented Mar 14, 2024

FedeDP commented Mar 14, 2024

FedeDP commented Mar 25, 2024

FedeDP Mar 25, 2024

FedeDP Mar 25, 2024

FedeDP Mar 25, 2024

FedeDP Mar 25, 2024

FedeDP Mar 25, 2024

FedeDP Mar 25, 2024

FedeDP Mar 25, 2024

poiana commented Mar 28, 2024

new(pkg/driver): try to fetch kernel headers leveraging driverkit library when building drivers #476

new(pkg/driver): try to fetch kernel headers leveraging driverkit library when building drivers #476

Conversation

FedeDP commented Mar 13, 2024

FedeDP commented Mar 14, 2024

FedeDP commented Mar 14, 2024

FedeDP commented Mar 14, 2024

FedeDP commented Mar 14, 2024

FedeDP commented Mar 25, 2024

FedeDP Mar 25, 2024

Choose a reason for hiding this comment

FedeDP Mar 25, 2024

Choose a reason for hiding this comment

FedeDP Mar 25, 2024

Choose a reason for hiding this comment

FedeDP Mar 25, 2024

Choose a reason for hiding this comment

FedeDP Mar 25, 2024

Choose a reason for hiding this comment

FedeDP Mar 25, 2024

Choose a reason for hiding this comment

FedeDP Mar 25, 2024

Choose a reason for hiding this comment

poiana commented Mar 28, 2024