docs(Advanced Process execution): /bin/busybox is not executed by id

[yukinakanaka]

What happened?

The document in Advanced Process execution says file_arg should be /bin/busybox. But in my environment, it was /usr/bin/id.

{
  "process_kprobe": {
    "process": {
      "exec_id": "dGV0cmFnb24tZGV2LWNvbnRyb2wtcGxhbmU6MjYyMzk2NTQyMjk3MTg6OTI4MTI=",
      "pid": 92812,
      "uid": 0,
      "cwd": "/",
      "binary": "/bin/bash",
      "flags": "execve",
      "start_time": "2024-04-15T05:13:06.794263737Z",
      "auid": 4294967295,
      "pod": {
        "namespace": "default",
        "name": "xwing",
        "container": {
          "id": "containerd://78478310306eaccb44637b5cba0d13663e44872cd403aabafec91c7c55a03517",
          "name": "spaceship",
          "image": {
            "id": "quay.io/cilium/json-mock@sha256:5aad04835eda9025fe4561ad31be77fd55309af8158ca8663a72f6abb78c2603",
            "name": "sha256:56b43d7e51feffe637c2837a8c70da02be98a51099533f288c78fa369f5c6991"
          },
          "start_time": "2024-04-14T09:16:49Z",
          "pid": 27
        },
        "pod_labels": {
          "app.kubernetes.io/name": "xwing",
          "class": "xwing",
          "org": "alliance"
        },
        "workload": "xwing",
        "workload_kind": "Pod"
      },
      "docker": "78478310306eaccb44637b5cba0d136",
      "parent_exec_id": "dGV0cmFnb24tZGV2LWNvbnRyb2wtcGxhbmU6MjYyMzQyMzYzMzY1NDA6OTI3Nzc=",
      "refcnt": 1,
      "tid": 92812
    },
    "parent": {
      "exec_id": "dGV0cmFnb24tZGV2LWNvbnRyb2wtcGxhbmU6MjYyMzQyMzYzMzY1NDA6OTI3Nzc=",
      "pid": 92777,
      "uid": 0,
      "cwd": "/",
      "binary": "/bin/bash",
      "flags": "execve rootcwd clone",
      "start_time": "2024-04-15T05:13:01.376370725Z",
      "auid": 4294967295,
      "pod": {
        "namespace": "default",
        "name": "xwing",
        "container": {
          "id": "containerd://78478310306eaccb44637b5cba0d13663e44872cd403aabafec91c7c55a03517",
          "name": "spaceship",
          "image": {
            "id": "quay.io/cilium/json-mock@sha256:5aad04835eda9025fe4561ad31be77fd55309af8158ca8663a72f6abb78c2603",
            "name": "sha256:56b43d7e51feffe637c2837a8c70da02be98a51099533f288c78fa369f5c6991"
          },
          "start_time": "2024-04-14T09:16:49Z",
          "pid": 27
        },
        "pod_labels": {
          "app.kubernetes.io/name": "xwing",
          "class": "xwing",
          "org": "alliance"
        },
        "workload": "xwing",
        "workload_kind": "Pod"
      },
      "docker": "78478310306eaccb44637b5cba0d136",
      "parent_exec_id": "dGV0cmFnb24tZGV2LWNvbnRyb2wtcGxhbmU6MjYyMzQyMTY0NzQ4NTE6OTI3Njc=",
      "tid": 92777
    },
    "function_name": "security_bprm_creds_from_file",
    "args": [
      {
        "file_arg": {
          "path": "/usr/bin/id",
          "permission": "-rwxr-xr-x"
        }
      }
    ],
    "action": "KPROBE_ACTION_POST",
    "policy_name": "process-exec-elf-begin",
    "return_action": "KPROBE_ACTION_POST"
  },
  "node_name": "tetragon-dev-control-plane",
  "time": "2024-04-15T05:13:06.795605779Z"
}

Tetragon Version

CLI version: v1.1.0-pre.0-779-g429672e8b

Kernel Version

Linux lima-tetragon-dev 5.15.0-102-generic #112-Ubuntu SMP Tue Mar 5 16:49:56 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

Kubernetes Version

Client Version: v1.29.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.2

Bugtool

No response

Relevant log output

• Docker image

kubectl get pods xwing -o jsonpath='{.spec.containers[0].image}'

quay.io/cilium/json-mock:v1.3.8@sha256:5aad04835eda9025fe4561ad31be77fd55309af8158ca8663a72f6abb78c2603

Anything else?

No response

[yukinakanaka]

It would be better to change id to another binary that has a symbolic link like /usr/bin/id -> /bin/busybox.

• Here are symbolic links in /bin of xwing in my environment.

root@xwing:/bin# find /bin/ -type l
/bin/captoinfo
/bin/sg
/bin/nawk
/bin/md5sum.textutils
/bin/dnsdomainname
/bin/ld.so
/bin/nisdomainname
/bin/domainname
/bin/linux64
/bin/sh
/bin/reset
/bin/ypdomainname
/bin/rbash
/bin/which
/bin/pager
/bin/awk
/bin/lastb
/bin/linux32
/bin/pidof
/bin/infotocap
/bin/ctstat
/bin/rtstat

• In the above binaries, awk is the most famous popular binary. So, how about using awk in the doc 🤔 ?

• /bin/awk points /usr/bin/mawk.

root@xwing:/bin# readlink -f awk
/usr/bin/mawk

• And a kprobe event of xwing in my environment was like this:

{
  "process_kprobe": {
    "process": {
      "exec_id": "dGV0cmFnb24tZGV2LWNvbnRyb2wtcGxhbmU6Mjk2MDc3ODA3Mjk2MTQ6MTA0MzE4",
      "pid": 104318,
      "uid": 0,
      "cwd": "/",
      "binary": "/bin/bash",
      "flags": "execve",
      "start_time": "2024-04-15T06:09:14.920763757Z",
      "auid": 4294967295,
      "pod": {
        "namespace": "default",
        "name": "xwing",
        "container": {
          "id": "containerd://78478310306eaccb44637b5cba0d13663e44872cd403aabafec91c7c55a03517",
          "name": "spaceship",
          "image": {
            "id": "quay.io/cilium/json-mock@sha256:5aad04835eda9025fe4561ad31be77fd55309af8158ca8663a72f6abb78c2603",
            "name": "sha256:56b43d7e51feffe637c2837a8c70da02be98a51099533f288c78fa369f5c6991"
          },
          "start_time": "2024-04-14T09:16:49Z",
          "pid": 34
        },
        "pod_labels": {
          "app.kubernetes.io/name": "xwing",
          "class": "xwing",
          "org": "alliance"
        },
        "workload": "xwing",
        "workload_kind": "Pod"
      },
      "docker": "78478310306eaccb44637b5cba0d136",
      "parent_exec_id": "dGV0cmFnb24tZGV2LWNvbnRyb2wtcGxhbmU6Mjg2MjgwOTg0MzYyMDI6MTAxMTY1",
      "refcnt": 1,
      "tid": 104318
    },
    "parent": {
      "exec_id": "dGV0cmFnb24tZGV2LWNvbnRyb2wtcGxhbmU6Mjg2MjgwOTg0MzYyMDI6MTAxMTY1",
      "pid": 101165,
      "uid": 0,
      "cwd": "/",
      "binary": "/bin/bash",
      "flags": "execve rootcwd clone",
      "start_time": "2024-04-15T05:52:55.238469637Z",
      "auid": 4294967295,
      "pod": {
        "namespace": "default",
        "name": "xwing",
        "container": {
          "id": "containerd://78478310306eaccb44637b5cba0d13663e44872cd403aabafec91c7c55a03517",
          "name": "spaceship",
          "image": {
            "id": "quay.io/cilium/json-mock@sha256:5aad04835eda9025fe4561ad31be77fd55309af8158ca8663a72f6abb78c2603",
            "name": "sha256:56b43d7e51feffe637c2837a8c70da02be98a51099533f288c78fa369f5c6991"
          },
          "start_time": "2024-04-14T09:16:49Z",
          "pid": 34
        },
        "pod_labels": {
          "app.kubernetes.io/name": "xwing",
          "class": "xwing",
          "org": "alliance"
        },
        "workload": "xwing",
        "workload_kind": "Pod"
      },
      "docker": "78478310306eaccb44637b5cba0d136",
      "parent_exec_id": "dGV0cmFnb24tZGV2LWNvbnRyb2wtcGxhbmU6Mjg2MjgwNzc1MjA3MzI6MTAxMTUy",
      "tid": 101165
    },
    "function_name": "security_bprm_creds_from_file",
    "args": [
      {
        "file_arg": {
          "path": "/usr/bin/mawk",
          "permission": "-rwxr-xr-x"
        }
      }
    ],
    "action": "KPROBE_ACTION_POST",
    "policy_name": "process-exec-elf-begin",
    "return_action": "KPROBE_ACTION_POST"
  },
  "node_name": "tetragon-dev-control-plane",
  "time": "2024-04-15T06:09:14.921128133Z"
}

• file_arg.path is /usr/bin/mawk, so looks good 😄 ?

[yukinakanaka]

Or, Is a script scenario better because we can handle all?

• A script is like:

cat << EOF > script.sh
#!/bin/sh
id
EOF

• When script.sh is executed, you can see /usr/bin/id not script.sh in event's args.file_arg.path because the process-exec-elf-begin tracing policy will report the final ELF or flat binary.

[mtardy]

Thanks for the detailed report!

1. In your first situation you saw that the event process.binary isn't resolved on our side.
2. In your second situation you saw that the security_bprm_creds_from_file hook resolves the file.
3. In your third situation it's not a symlink, it's just that /bin/sh will call exec with id thus explaining why you should see two exec event, one for /bin/sh and one for /usr/bin/id.

Indeed in the events we report the symbolic links and not the final binary, this is something I bumped into when reimplementing the matchBinaries selectors as I think it matches the real underlying file and not the symlink displayed in the event. Here is what I wrote back then:

Previously, matchBinaries reacted on execve filename, which is relative and potentially a symlink. In the event, we return the absolute path (we do the resolution in userspace, potentially combining filename + cwd). But it can still be symlink. For example I get /usr/sbin/iptables in events while the real binary, when reading the proc exe, is /usr/sbin/xtables-nft-multi.

We now read the proc exe with the new matchBinaries implem, I also resolve all the symlinks (because of exe nature). Problem: if a user writes /usr/sbin/iptables in the matchBinaries of a tracing policy, it will never match because the real exe is /usr/sbin/xtables-nft-multi.

My (bad) idea was to automatically, in the back of the user, resolve symlinks in tracing policies' matchBinaries, but this can fail: if the binary pointed by the matchBinaries doesn't exist when I parse the TracingPolicy, I have no way of resolving any symlink.

And another "problem" is that we do not resolve the symlink when reporting the event to the user, meaning user will see /usr/sbin/iptables in the event, not the "real" binary.

So it's mostly an open question on whether we want to report the symlink or the resolved path to the user in process.binary.

[yukinakanaka]

Thank you for your comment!

I may not understand your comment correctly, but I just wanted to report that I'm unable to replicate the ProcessKprobe events described in the documentation on my environment.

The args.file_arg.path of the doc is /bin/busybox, but in my environment, it was /use/bin/id even though I used the same the process-exec-elf-begin tracing policy and the same xwing Pod.

The doc's event

My environment's event

Could you try to see what event you get by running id on xwing in your environment?

(I thought the documentation was outdated, so I proposed other commands in the above second and third comments. But please ignore them.)