Falco keeps restarting when run with ebpf driver in least privilege mode

[Nnoromuche]

Describe the bug
When I run falco as a demonset in least priviledge mode using the edpf driver on a red hat enterprise linux 8.9, I see multiple falco process restarts in the logs on a kubernetes environment. When I run falco with the edpf driver in privilege mode, it works fine, however running in least privilege results in multiple restarts.

How to reproduce it
Add the following capabilities to the security context in the deployment.yaml file

securityContext: 
   capabilities: 
      add: 
      - SYS_ADMIN 
      - SYS_RESOURCE 
      - SYS-PTRACE 

Logs shows

SUPERVISOR: 2024/02/20 16:43:12 Creating new supervisor for falco
SUPERVISOR: 2024/02/20 16:43:12 Creating new supervisor for falco-exporter
SUPERVISOR: 2024/02/20 16:43:12 Creating new supervisor for falcosidekick
SUPERVISOR: 2024/02/20 16:43:12 Starting reaper
SUPERVISOR: 2024/02/20 16:43:12 Running `/usr/bin/run.sh`
2024-02-20 16:43:12 INFO  Running falcoctl driver config
                      ├ name: falco
                      ├ version: 7.0.0+driver
                      ├ type: ebpf
                      ├ host-root: /host
                      └ repos: https://download.falco.org/driver
2024-02-20 16:43:12 WARN  Avoid updating Falco configuration
                      ├ config: /etc/falco/falco.yaml
                      └ reason: engine.kind is not driver driven: 
2024-02-20 16:43:12 INFO  Running falcoctl driver install
                      ├ driver version: 7.0.0+driver
                      ├ driver type: ebpf
                      ├ driver name: falco
                      ├ compile: true
                      ├ download: true
                      ├ arch: x86_64
                      ├ kernel release: 4.18.0-513.11.1.el8_9.x86_64
                      └ kernel version: #1 SMP Thu Dec 7 03:06:13 EST 2023
2024-02-20 16:43:12 INFO  Found distro target: rhel
2024-02-20 16:43:12 INFO  Removing eBPF probe symlink
                      └ path: /root/.falco/falco-bpf.o
2024-02-20 16:43:12 INFO  Trying to download a driver.
                      └ url: https://download.falco.org/driver/7.0.0%2Bdriver/x86_64/falco_rhel_4.18.0-513.11.1.el8_9.x86_64_1.o
2024-02-20 16:43:13 WARN  Non-200 response from url. code: 404
2024-02-20 16:43:13 WARN  unable to find a prebuilt driver 
2024-02-20 16:43:13 INFO  Mounting debugfs for bpf driver. 
2024-02-20 16:43:30 INFO  Copying built driver to its destination.
                      ├ src: /usr/src/falco-7.0.0+driver/bpf/probe.o
                      └ dst: /root/.falco/7.0.0+driver/x86_64/falco_rhel_4.18.0-513.11.1.el8_9.x86_64_1.o
2024-02-20 16:43:30 INFO  Driver built.
                      └ path: /root/.falco/7.0.0+driver/x86_64/falco_rhel_4.18.0-513.11.1.el8_9.x86_64_1.o
2024-02-20 16:43:30 INFO  Symlinking eBPF probe
                      ├ src: /root/.falco/7.0.0+driver/x86_64/falco_rhel_4.18.0-513.11.1.el8_9.x86_64_1.o
                      └ dest: /root/.falco/falco-bpf.o
2024-02-20 16:43:30 INFO  eBPF probe symlinked 
SUPERVISOR: 2024/02/20 16:43:30 Completed `/usr/bin/run.sh`
SUPERVISOR: 2024/02/20 16:43:30 Starting falco
SUPERVISOR: 2024/02/20 16:43:30 falco started successfully
SUPERVISOR: 2024/02/20 16:43:30 Starting falco-exporter
SUPERVISOR: 2024/02/20 16:43:30 falco-exporter started successfully
SUPERVISOR: 2024/02/20 16:43:30 Starting falcosidekick
Events detected: 0
Rule counts by severity:
Triggered rules by rule name:
SUPERVISOR: 2024/02/20 16:43:31 falcosidekick started successfully
SUPERVISOR: 2024/02/20 16:43:41 falco process has stopped, restarting.
Events detected: 0
Rule counts by severity:
Triggered rules by rule name:
SUPERVISOR: 2024/02/20 16:43:51 falco process has stopped, restarting.
Events detected: 0
Rule counts by severity:
Triggered rules by rule name:
SUPERVISOR: 2024/02/20 16:44:01 falco process has stopped, restarting.
Events detected: 0
Rule counts by severity:
Triggered rules by rule name:
SUPERVISOR: 2024/02/20 16:44:11 falco process has stopped, restarting.
Events detected: 0
Rule counts by severity:
Triggered rules by rule name:
SUPERVISOR: 2024/02/20 16:44:21 falco process has stopped, restarting.

Expected behaviour

I will like falco to run in least priviledge mode on a red hat enterprise linux 8.9 machine without multiple restarts.

Environment

• Falco version: 0.38
• OS: Rhel
• Installation method: Deployment.yaml in a kubernetes environment

[rashim27us]

@Nnoromuche There are some possible reasons for this issue, in my case when I was running Falco in the least privileged mode it didn't have the necessary permissions to load the eBPF driver. If we use Pod Security policies in the Kubernetes cluster, they might restrict the capabilities of the Falco pods, preventing them from loading the eBPF driver.

Here are the key permissions required to run Falco with eBPF driver:

1. SYS_ADMIN_Capability
2. Host PID Namespace
3. Host Networking
4. Volume Mounts

You can check whether these necessary permissions are given or not in Kubernetes least privileged mode.

[Nnoromuche]

@rashim27us thanks, I am running falco with the below deployment.yaml file and the SYS_ADMIN_Capability is being set.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: test-falco
  namespace: test
spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 10%
  selector:
    matchLabels:
      app: test-falco
  template:
    metadata:
      name: test-falco
      namespace: test
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/path: "/metrics"
        prometheus.io/port: "12076"
        prometheus.istio.io/merge-metrics: "false"
        sidecar.istio.io/inject: "false"
      labels:
        app: test-falco
        addonmanager.kubernetes.io/mode: Reconcile
    spec:
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      tolerations:
      - operator: Exists
      volumes:
      - name: dshm
        emptyDir:
          medium: Memory
      - name: containerd-socket
        hostPath:
          path: /run/containerd/containerd.sock
      - name: proc-fs
        hostPath:
          path: /proc
      - name: boot-fs
        hostPath:
          path: /boot
      - name: lib-modules
        hostPath:
          path: /lib/modules
      - name: usr-fs
        hostPath:
          path: /usr
      - name: etc-fs
        hostPath:
          path: /etc
      - name: root-falco-fs
        emptyDir: {}
      containers:
      - name: test-falco
        image: (( concat "{{ FALCO_IMAGE_REPO_HOSTNAME}}/test/test-falco:" metadata.annotations.version ))
        env:
        - name: HOST_ROOT
          value: "/host"
        securityContext:
          capabilities:
            drop:
            - all
            add:
            - SYS_ADMIN
            - SYS_RESOURCE
            - SYS_PTRACE
        livenessProbe:
          httpGet:
            path: /liveness
            port: 12077
          periodSeconds: 60
          failureThreshold: 10
        readinessProbe:
          httpGet:
            path: /readiness
            port: 12077
          periodSeconds: 60
          failureThreshold: 10
        volumeMounts:
        - mountPath: /host/run/containerd/containerd.sock
          name: containerd-socket
        - mountPath: /host/proc
          name: proc-fs
          readOnly: true
        - mountPath: /host/boot
          name: boot-fs
          readOnly: true
        - mountPath: /host/lib/modules
          name: lib-modules
          readOnly: false
        - mountPath: /host/usr
          name: usr-fs
          readOnly: true
        - mountPath: /dev/shm
          name: dshm
        - mountPath: /host/etc
          name: etc-fs
          readOnly: true
        - mountPath: /root/.falco
          name: root-falco-fs
        resources:
          requests:
            cpu: 25m
            memory: 230Mi
          limits:
            memory: 400Mi

The edpf driver is installed using falcoctl commands

/usr/bin/falcoctl driver config --type ebpf
/usr/bin/falcoctl driver install --compile --download

Wondering is there a missing configuration in the above deployment.yaml file?