Add runc symlink for rancher rke2 #2764

matthyx · 2024-04-24T10:07:02Z

Add runc symlink for rancher rke2

Now that we support symlink resolution in runtime path, we can support Rancher rke2 out of the box (similar to what we have for k3s).

How to use

Install rancher https://docs.rke2.io/install/quickstart and try tracing without specifying any runc path.

Testing done

Compiled ig and successfully ran some tracers on rke2.

pkg/container-hook/tracer.go

pkg/runcfanotify/runcfanotify.go

eiffel-fl · 2024-04-24T12:59:34Z

pkg/runcfanotify/runcfanotify.go

+	"/usr/bin/conmon",
+	"/var/lib/rancher/k3s/data/current/bin/runc",


These changes should belong to a specific commit, as I think we forgot them in previous modifications?

I am also wondering if we should not unify these two arrays into one which would be used by these two files.

waiting for @alban and if OK will refactor

I already opened this PR with regard to refactoring:
#2766
In any case, we have several levels of refactoring and the code itself can be shared between the two packages, but I am not 100% sure for the arrays.

awesome, so I just close this one and you add the new symlinked path to your PR?

thanks for being reactive Francis!

awesome, so I just close this one and you add the new symlinked path to your PR?

Please, leave yours open for now as I cannot give you any ETA for when mine will be merged.

thanks for being reactive Francis!

You are welcome.

eiffel-fl

Hi!

I tested it locally but rke2 seems not able to use container-hook and/or runc-fanotify:

$ ./rke2
...
INFO[0131] Tunnel authorizer set Kubelet Port 10250 
# No events are reported with latest:
$ kubectl logs -n gadget gadget-ktwf6
time="2024-04-26T09:30:03Z" level=info msg="OS detected: Ubuntu 22.04.4 LTS"
time="2024-04-26T09:30:03Z" level=info msg="Kernel detected: 6.5.0-28-generic"
time="2024-04-26T09:30:03Z" level=info msg="Gadget Image: ghcr.io/inspektor-gadget/inspektor-gadget:latest"
...
time="2024-04-26T09:30:05Z" level=warning msg="Skip pod kube-system/rke2-snapshot-validation-webhook-54c5989b65-sg2jp: cannot find container (ID: containerd://0b7c4bb1015db08aef1943e3b136bd364663964bc1e14835fbfe19a058566bfc): loading container with id \"0b7c4bb1015db08aef1943e3b136bd364663964bc1e14835fbfe19a058566bfc\": container \"0b7c4bb1015db08aef1943e3b136bd364663964bc1e14835fbfe19a058566bfc\" in namespace \"k8s.io\": not found"
$ ./kubectl-gadget trace exec -A                          (remotes/matthyx/patch-2) *%
K8S.NODE          K8S.NAMESPACE     K8S.POD           K8S.CONTAINER     PID       PPID      COMM      PCOMM    RET ARGS                  
^C%
# Same with your patch:
$ kubectl logs -n gadget gadget-tndjt                      (remotes/matthyx/patch-2) %
time="2024-04-26T09:49:10Z" level=info msg="OS detected: Ubuntu 22.04.4 LTS"
time="2024-04-26T09:49:10Z" level=info msg="Kernel detected: 6.5.0-28-generic"
time="2024-04-26T09:49:10Z" level=info msg="Gadget Image: francisrkeregistry.azurecr.io/gadget:HEAD"
...
time="2024-04-26T09:49:11Z" level=warning msg="Skip pod kube-system/rke2-snapshot-validation-webhook-54c5989b65-sg2jp: cannot find container (ID: containerd://0b7c4bb1015db08aef1943e3b136bd364663964bc1e14835fbfe19a058566bfc): loading container with id \"0b7c4bb1015db08aef1943e3b136bd364663964bc1e14835fbfe19a058566bfc\": container \"0b7c4bb1015db08aef1943e3b136bd364663964bc1e14835fbfe19a058566bfc\" in namespace \"k8s.io\": not found"
$ ./kubectl-gadget trace exec -A                           (remotes/matthyx/patch-2) %
K8S.NODE          K8S.NAMESPACE     K8S.POD           K8S.CONTAINER     PID       PPID      COMM      PCOMM    RET ARGS                  
^C%

Am I missing something?

Best regards.

matthyx · 2024-04-26T09:55:01Z

Hi!

I tested it locally but rke2 seems not able to use container-hook and/or runc-fanotify:

$ ./rke2
...
INFO[0131] Tunnel authorizer set Kubelet Port 10250 
# No events are reported with latest:
$ kubectl logs -n gadget gadget-ktwf6
time="2024-04-26T09:30:03Z" level=info msg="OS detected: Ubuntu 22.04.4 LTS"
time="2024-04-26T09:30:03Z" level=info msg="Kernel detected: 6.5.0-28-generic"
time="2024-04-26T09:30:03Z" level=info msg="Gadget Image: ghcr.io/inspektor-gadget/inspektor-gadget:latest"
...
time="2024-04-26T09:30:05Z" level=warning msg="Skip pod kube-system/rke2-snapshot-validation-webhook-54c5989b65-sg2jp: cannot find container (ID: containerd://0b7c4bb1015db08aef1943e3b136bd364663964bc1e14835fbfe19a058566bfc): loading container with id \"0b7c4bb1015db08aef1943e3b136bd364663964bc1e14835fbfe19a058566bfc\": container \"0b7c4bb1015db08aef1943e3b136bd364663964bc1e14835fbfe19a058566bfc\" in namespace \"k8s.io\": not found"
$ ./kubectl-gadget trace exec -A                          (remotes/matthyx/patch-2) *%
K8S.NODE          K8S.NAMESPACE     K8S.POD           K8S.CONTAINER     PID       PPID      COMM      PCOMM    RET ARGS                  
^C%
# Same with your patch:
$ kubectl logs -n gadget gadget-tndjt                      (remotes/matthyx/patch-2) %
time="2024-04-26T09:49:10Z" level=info msg="OS detected: Ubuntu 22.04.4 LTS"
time="2024-04-26T09:49:10Z" level=info msg="Kernel detected: 6.5.0-28-generic"
time="2024-04-26T09:49:10Z" level=info msg="Gadget Image: francisrkeregistry.azurecr.io/gadget:HEAD"
...
time="2024-04-26T09:49:11Z" level=warning msg="Skip pod kube-system/rke2-snapshot-validation-webhook-54c5989b65-sg2jp: cannot find container (ID: containerd://0b7c4bb1015db08aef1943e3b136bd364663964bc1e14835fbfe19a058566bfc): loading container with id \"0b7c4bb1015db08aef1943e3b136bd364663964bc1e14835fbfe19a058566bfc\": container \"0b7c4bb1015db08aef1943e3b136bd364663964bc1e14835fbfe19a058566bfc\" in namespace \"k8s.io\": not found"
$ ./kubectl-gadget trace exec -A                           (remotes/matthyx/patch-2) %
K8S.NODE          K8S.NAMESPACE     K8S.POD           K8S.CONTAINER     PID       PPID      COMM      PCOMM    RET ARGS                  
^C%

Am I missing something?

Best regards.

can you log in debug and make sure we pick the right runc?

eiffel-fl · 2024-04-26T10:37:14Z

Hi!
I tested it locally but rke2 seems not able to use container-hook and/or runc-fanotify:

$ ./rke2
...
INFO[0131] Tunnel authorizer set Kubelet Port 10250 
# No events are reported with latest:
$ kubectl logs -n gadget gadget-ktwf6
time="2024-04-26T09:30:03Z" level=info msg="OS detected: Ubuntu 22.04.4 LTS"
time="2024-04-26T09:30:03Z" level=info msg="Kernel detected: 6.5.0-28-generic"
time="2024-04-26T09:30:03Z" level=info msg="Gadget Image: ghcr.io/inspektor-gadget/inspektor-gadget:latest"
...
time="2024-04-26T09:30:05Z" level=warning msg="Skip pod kube-system/rke2-snapshot-validation-webhook-54c5989b65-sg2jp: cannot find container (ID: containerd://0b7c4bb1015db08aef1943e3b136bd364663964bc1e14835fbfe19a058566bfc): loading container with id \"0b7c4bb1015db08aef1943e3b136bd364663964bc1e14835fbfe19a058566bfc\": container \"0b7c4bb1015db08aef1943e3b136bd364663964bc1e14835fbfe19a058566bfc\" in namespace \"k8s.io\": not found"
$ ./kubectl-gadget trace exec -A                          (remotes/matthyx/patch-2) *%
K8S.NODE          K8S.NAMESPACE     K8S.POD           K8S.CONTAINER     PID       PPID      COMM      PCOMM    RET ARGS                  
^C%
# Same with your patch:
$ kubectl logs -n gadget gadget-tndjt                      (remotes/matthyx/patch-2) %
time="2024-04-26T09:49:10Z" level=info msg="OS detected: Ubuntu 22.04.4 LTS"
time="2024-04-26T09:49:10Z" level=info msg="Kernel detected: 6.5.0-28-generic"
time="2024-04-26T09:49:10Z" level=info msg="Gadget Image: francisrkeregistry.azurecr.io/gadget:HEAD"
...
time="2024-04-26T09:49:11Z" level=warning msg="Skip pod kube-system/rke2-snapshot-validation-webhook-54c5989b65-sg2jp: cannot find container (ID: containerd://0b7c4bb1015db08aef1943e3b136bd364663964bc1e14835fbfe19a058566bfc): loading container with id \"0b7c4bb1015db08aef1943e3b136bd364663964bc1e14835fbfe19a058566bfc\": container \"0b7c4bb1015db08aef1943e3b136bd364663964bc1e14835fbfe19a058566bfc\" in namespace \"k8s.io\": not found"
$ ./kubectl-gadget trace exec -A                           (remotes/matthyx/patch-2) %
K8S.NODE          K8S.NAMESPACE     K8S.POD           K8S.CONTAINER     PID       PPID      COMM      PCOMM    RET ARGS                  
^C%

Am I missing something?
Best regards.

can you log in debug and make sure we pick the right runc?

Nevermind! We indeed pick it:

time="2024-04-26T10:28:33Z" level=debug msg="container-hook: trying runtime at /host/var/lib/rancher/rke2/data/v1.29.2-rke2r1-8bfebc2d9089/bin/runc"
time="2024-04-26T10:28:33Z" level=debug msg="container-hook: monitoring runtime at /host/var/lib/rancher/rke2/data/v1.29.2-rke2r1-8bfebc2d9089/bin/runc"

I can trace newly added pod with your changes:

$ kubectl run --restart=Never -ti --image=busybox mypod -- sh -c 'while /bin/true ; do whoami ; sleep 3 ; done'
If you don't see a command prompt, try pressing enter.
root
root
root
...
$ ./kubectl-gadget trace exec                              (remotes/matthyx/patch-2) %
K8S.NODE          K8S.NAMESPACE     K8S.POD           K8S.CONTAINER     PID       PPID      COMM      PCOMM    RET ARGS                  
pwmachine         default           mypod             mypod             129879    129434    true      sh       0   /bin/true             
pwmachine         default           mypod             mypod             129880    129434    whoami    sh       0   /bin/whoami           
pwmachine         default           mypod             mypod             129942    129434    true      sh       0   /bin/true             
pwmachine         default           mypod             mypod             129943    129434    whoami    sh       0   /bin/whoami

But not previous one:

$ ./kubectl-gadget trace exec -n kube-system
K8S.NODE          K8S.NAMESPACE     K8S.POD           K8S.CONTAINER     PID       PPID      COMM      PCOMM    RET ARGS
# Nothing is printed

matthyx · 2024-04-26T10:47:09Z

But not previous one:

$ ./kubectl-gadget trace exec -n kube-system
K8S.NODE          K8S.NAMESPACE     K8S.POD           K8S.CONTAINER     PID       PPID      COMM      PCOMM    RET ARGS
# Nothing is printed

I wonder if the system pods are running from a different runc?

eiffel-fl · 2024-04-26T11:09:01Z

But not previous one:

$ ./kubectl-gadget trace exec -n kube-system
K8S.NODE          K8S.NAMESPACE     K8S.POD           K8S.CONTAINER     PID       PPID      COMM      PCOMM    RET ARGS
# Nothing is printed

I wonder if the system pods are running from a different runc?

Maybe,

But not previous one:

$ ./kubectl-gadget trace exec -n kube-system
K8S.NODE          K8S.NAMESPACE     K8S.POD           K8S.CONTAINER     PID       PPID      COMM      PCOMM    RET ARGS
# Nothing is printed

I wonder if the system pods are running from a different runc?

Maybe, but I am rather wondering if this is a not a timing problem.
As I cannot trace the gadget namespace too.
Any idea @alban?

alban · 2024-04-29T15:57:33Z

pkg/runcfanotify/runcfanotify.go

+			runcPath, err := securejoin.SecureJoin(host.HostRoot, r)
+			if err != nil {
+				log.Debugf("Runcfanotify: securejoin failed: %s", err)


In some cases, the error will already contain "SecureJoin":
https://github.com/cyphar/filepath-securejoin/blob/v0.2.4/join.go#L58
https://cs.opensource.google/go/go/+/refs/tags/go1.22.2:src/io/fs/fs.go;l=256

But not in all cases. So I wonder if we should repeat "securejoin" here. What do you think?

this is matching what we have in pkg/container-hook/tracer.go

matthyx · 2024-05-24T15:58:15Z

@alban can we have a look at merging this, please?

eiffel-fl · 2024-05-27T08:11:08Z

Hi!

@alban can we have a look at merging this, please?

I would like to first merge #2766 as it will ease merging this one.

Best regards.

eiffel-fl · 2024-05-28T12:26:30Z

Hi!

I just merged #2766, can you please rebase?
Your commit should now just be a matter of adding the symlink here:

inspektor-gadget/pkg/container-hook/runtime-finder/finder.go

Lines 34 to 44 in 54fe2ac

    
           var RuntimePaths = []string{ 
        
           	"/bin/runc", 
        
           	"/usr/bin/runc", 
        
           	"/usr/sbin/runc", 
        
           	"/usr/local/bin/runc", 
        
           	"/usr/local/sbin/runc", 
        
           	"/usr/lib/cri-o-runc/sbin/runc", 
        
           	"/run/torcx/unpack/docker/bin/runc", // Used in Flatcar Container Linux 
        
           	"/usr/bin/crun", 
        
           	"/var/lib/rancher/k3s/data/current/bin/runc", // Used in k3s 
        
           }

Best regards.

Signed-off-by: Matthias Bertschy <matthias.bertschy@gmail.com>

matthyx · 2024-05-28T13:42:31Z

thanks @eiffel-fl done!

alban

LGTM

eiffel-fl

I tested it again and it works fine:

$ ./kubectl-gadget trace exec                             (remotes/matthyx/patch-2) *%
K8S.NODE          K8S.NAMESPACE     K8S.POD           K8S.CONTAINER     PID       PPID      COMM      PCOMM    RET ARGS                  
pwmachine         default           test-pod          test-pod          212820    51797     ls        bash     0   /usr/bin/ls
$ ../kubectl get node | awk '{ print $5 }'                (remotes/matthyx/patch-2) *%
VERSION
v1.29.2+rke2r1

Thank you for this contribution!

matthyx requested a review from alban as a code owner April 24, 2024 10:07

alban reviewed Apr 24, 2024

View reviewed changes

pkg/container-hook/tracer.go Outdated Show resolved Hide resolved

matthyx force-pushed the patch-2 branch from 88f8e19 to 72ec2b4 Compare April 24, 2024 10:57

eiffel-fl reviewed Apr 24, 2024

View reviewed changes

pkg/runcfanotify/runcfanotify.go Outdated Show resolved Hide resolved

eiffel-fl reviewed Apr 24, 2024

View reviewed changes

eiffel-fl reviewed Apr 26, 2024

View reviewed changes

matthyx force-pushed the patch-2 branch from 72ec2b4 to b741327 Compare April 29, 2024 15:35

alban reviewed Apr 29, 2024

View reviewed changes

alban mentioned this pull request Apr 30, 2024

container-hook: map gets full and it stops working #2789

Closed

alban mentioned this pull request May 13, 2024

pkg/{runcfanotify, container-hook}: Refactor code to use notify package. #2766

Merged

matthyx force-pushed the patch-2 branch from b741327 to 6d3ecf4 Compare May 24, 2024 15:56

Add runc symlink for rancher rke2

bf83e22

Signed-off-by: Matthias Bertschy <matthias.bertschy@gmail.com>

matthyx force-pushed the patch-2 branch from 6d3ecf4 to bf83e22 Compare May 28, 2024 13:41

alban approved these changes May 28, 2024

View reviewed changes

eiffel-fl approved these changes May 28, 2024

View reviewed changes

eiffel-fl merged commit 5287634 into inspektor-gadget:main May 28, 2024
57 checks passed

matthyx deleted the patch-2 branch May 31, 2024 05:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add runc symlink for rancher rke2 #2764

Add runc symlink for rancher rke2 #2764

matthyx commented Apr 24, 2024

eiffel-fl Apr 24, 2024

eiffel-fl Apr 24, 2024

matthyx Apr 24, 2024

eiffel-fl Apr 24, 2024

matthyx Apr 24, 2024

matthyx Apr 24, 2024

eiffel-fl Apr 24, 2024

eiffel-fl left a comment

matthyx commented Apr 26, 2024

eiffel-fl commented Apr 26, 2024

matthyx commented Apr 26, 2024

eiffel-fl commented Apr 26, 2024

alban Apr 29, 2024

matthyx May 24, 2024

matthyx commented May 24, 2024

eiffel-fl commented May 27, 2024

eiffel-fl commented May 28, 2024

matthyx commented May 28, 2024

alban left a comment

eiffel-fl left a comment

		"/usr/bin/conmon",
		"/var/lib/rancher/k3s/data/current/bin/runc",

Add runc symlink for rancher rke2 #2764

Add runc symlink for rancher rke2 #2764

Conversation

matthyx commented Apr 24, 2024

Add runc symlink for rancher rke2

How to use

Testing done

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eiffel-fl left a comment

Choose a reason for hiding this comment

matthyx commented Apr 26, 2024

eiffel-fl commented Apr 26, 2024

matthyx commented Apr 26, 2024

eiffel-fl commented Apr 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matthyx commented May 24, 2024

eiffel-fl commented May 27, 2024

eiffel-fl commented May 28, 2024

matthyx commented May 28, 2024

alban left a comment

Choose a reason for hiding this comment

eiffel-fl left a comment

Choose a reason for hiding this comment