eBPF

[themighty1]

Happy new year one and all !!!
@gustavo-iniguez-goya , have you done any coding towards using eBPF in Opensnitch?

Should be easy to implement.
Here's a small prog from
https://github.com/iovisor/bcc/blob/master/examples/tracing/tcpv4connect.py
gets access to kernel struck sock from which sport, dport, saddr, daddr can be obtained.

[gustavo-iniguez-goya]

Happy new year! 🎉

nope, I tried but on debian I was unable to compile "complex" programs, outside of the kernel tree (due to headers import errors).
Others use ubuntu18 to compile them.

Using bp2go looks trivial to use:
https://github.com/lmb/ship-bpf-with-go

[themighty1]

I created a branch for ebpf:
https://github.com/themighty1/opensnitch/commits/ebpf_new

The commit message contains info on how to build the dependencies and to run.
It works with TCP4 and UDP4

A few observations:

1. Sometimes (~10%) the packet from nfqueue arrives faster than data from ebpf. In such cases we have to wait 1-3 usecs before data arrives from ebpf. The max delay that I've seen is 20 usec

2. I get very many connections from nfqueue which are not found in ebpf. Those connection have the same sport/saddr/daddr/dport as other connections which were made earlier and found in ebpf. Those are duplicate connections. This needs investigation.

3. occasionally ebpf report connections which never come from nfqueue. Usually those are weird connections like 0.0.0.0:0

I intend to investigate those odd behaviours as time permits.

[gustavo-iniguez-goya]

superb @themighty1 ! 🎉

Some questions and considerations:

Sometimes (~10%) the packet from nfqueue arrives faster than data from ebpf.

We could use in a near future XDP to drop packets, and not depend on nfqueue.

themighty1@f45c3c3#diff-15d7687250ed7ebda3aa02642d66c977276676567dc914a625eb47f5185fcfbcR87

Since we get the PID from eBPF, is it necessary to lookup the inode? We should go directly to /proc/<PID>/ parse all the info and send it to the UI.

kprobe__

Does the kprobes depend on having mounted debugfs? It's something I have not very clear. I guess no.

What Ubuntu version did you use?

[themighty1]

Sometimes (~10%) the packet from nfqueue arrives faster than data from ebpf.

We could use in a near future XDP to drop packets, and not depend on nfqueue.

Yes. I agree. Right now I'm seeing that (strangely) ebpf is sometimes slower than nfqueue.

Since we get the PID from eBPF, is it necessary to lookup the inode? We should go directly to /proc/<PID>/ parse all the info and send it to the UI.

I only left inode lookup for debug purposes, to investigate duplicate connections.

Does the kprobes depend on having mounted debugfs? It's something I have not very clear. I guess no.

Yes, it depends on it. I just unmounted mine and got lots of errors running this branch.

What Ubuntu version did you use?

I use 18.04

[gustavo-iniguez-goya]

hey, now that v1.3.5 is out, I wanted to test this branch, but it looks that it's gone. Could we work on it? I think even if we depend on debugfs, it's a better approach than parsing trace_pipe

[themighty1]

@gustavo-iniguez-goya , hi.
Sorry, here's a new branch https://github.com/themighty1/opensnitch/tree/ebpf_onmaster
It does not work right off the bat, I amin the middle if reshuffling some code, commenting it out, etc.

[gustavo-iniguez-goya]

ok, no problem. I want to finish some things first, but definitely I'd like to test it.

[themighty1]

Huh, I switched to a Ubuntu 20.04 machine and I cannot compile the ebpf branch anymore. I get a boatload of kernel header errors. Sad. I'll try to chroot into Ubuntu 18.

[themighty1]

I was able to get rid of the errors by removing ubuntu's bcc lib and installing bcc from source
https://github.com/iovisor/bcc/blob/master/INSTALL.md#ubuntu---source

[themighty1]

I just updated https://github.com/themighty1/opensnitch/tree/ebpf_onmaster

It is in a more polished state, works with tcp/udp/tcp6/udp6
Average time to lookup PID for a new connection is 20usec.
I have not had any stray connections after suspend/resume.
If a connection was not found in ebpf and netlink reports UID==0 then it is assumed to be an in-kernel connection.

The code will exit if it encounters a connection from netfilter queue which is not accounted for.

[gustavo-iniguez-goya]

👍 ok, ubuntu18 installed, bcc libs compiled, and opensnitchd compiled. I'll test it soon.

I guess that if we commit this, I'd need a ubuntu18 to build the packages. It worries me if we'll be able to compile for multiple architectures as we do now. Or maybe install the bcc libs inside the pbuilder chroot environments.. no idea.

[themighty1]

fwiw, I am running this on Ubuntu 20 with no compilation problems.

[gustavo-iniguez-goya]

The thing is if we use a platform like build.opensuse.org or launchpad.net to build the packages, the compilation and installation of the bcc libs should be integrated as part of the build process, no?

By the way, Transmission and probably others apps close connections without waiting to finish. This causes some weird behaviour, from not being able to get the pid to not even know that there was an attempt to establish a connection. I mimicked this behaviour with this simple prog:
c-client-async.c.txt

This prog causes to exit in this line: https://github.com/themighty1/opensnitch/blob/ebpf_onmaster/daemon/conman/connection.go#L143

[themighty1]

@gustavo-iniguez-goya , yes the bcc lib should be built as part of the build process.

Thanks for pointing out that offending line with os.Exit(). You can just comment it out.

In your C code the app quits immediately. But in the case of Transmission it keeps running even after it closes the socket immediately, so Transmission should be found by its PID.
Have you seen Transmission hitting that os.Exit() codepath?

[gustavo-iniguez-goya]

Testing it now, it's working fine so far.

On the other hand, we should also distribute the libcc because in Debian for example it's not packaged:

$ ldd opensnitchd 
	linux-vdso.so.1 (0x00007ffc19d80000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fbede342000)
	libnetfilter_queue.so.1 => /usr/lib/x86_64-linux-gnu/libnetfilter_queue.so.1 (0x00007fbede338000)
	libnfnetlink.so.0 => /usr/lib/x86_64-linux-gnu/libnfnetlink.so.0 (0x00007fbede131000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fbede10f000)
	libbcc.so.0 => not found

$ dpkg -S libbcc.so.0
dpkg-query: no path found matching pattern *libbcc.so.0*

If instead of kprobes we'd use ebpf programs we could distribute just the .o object file.

[themighty1]

From this post https://facebookmicrosites.github.io/bpf/blog/2020/02/19/bpf-portability-and-co-re.html
my takeaway is that .o is not portable, only bcc project provides portability.
CO-RE will make .o portable but it is still wip.

[themighty1]

I made a new branch https://github.com/themighty1/opensnitch/tree/newbpf
where we do not rely on bcc anymore but instead compile an .o file. All the steps are in the commit message.

I was not correct to say that .o is not portable. It is portable to a degree.
I created a .o file for opensnitch on 5.8 kernel and used it successfully on kernel 5.4. However it failed on kernel 4.15

[gustavo-iniguez-goya]

I've finally compiled and tested it. Flawlessly using the instructions of the commit, awesome! (sorry for the delay...)

So the only thing left is testing it to get an idea where it'll work (just out of curiosity). As it's a eBPF module there's no risk of freeze the system if it doesn't load or fail.

We still need to have debugfs mounted:

kprobe/tcp_v4_connect
panic: cannot open kprobe_events: open /sys/kernel/debug/tracing/kprobe_events: no such file or directory

goroutine 1 [running]:
github.com/evilsocket/opensnitch/daemon/ebpf.Start(0x45d401, 0xc0011d5d00)
	/home/ga/opensnitch/daemon/ebpf/ebpf.go:87 +0x14a9
github.com/evilsocket/opensnitch/daemon/procmon/monitor.Init()
	/home/ga/opensnitch/daemon/procmon/monitor/init.go:18 +0x35

For now opensnitch.o tested on Debian Sid, kernel 5.10, and working fine.

[gustavo-iniguez-goya]

Ok, back with this again @themighty1 . Can we tidy everything up and remove all the debug Println() towards pushing a PR?

Some suggestions:

• move daemon/ebpf/ to daemon/procmon/ebpf/ since it's just another monitor method (we should also create ftrace/ and move watcher.go there some day).
• move this block to daemon/procmon/ebpf/ and just add here procmon.GetPidFromeBPF() or something similar:
  https://github.com/themighty1/opensnitch/blob/fedf0e1c90aaefe9eb1585c076c994328ff6cbf1/daemon/conman/connection.go#L93
• get the UID from ebpf to avoid query the kernel for it: ebpf.GetPid() int{} -> ebpf.GetPid() (pid, uid int){}

I can help with these changes.

[themighty1]

hi, @gustavo-iniguez-goya , I've been stress-testing my code and found corner-cases when connections were not picked-up by ebpf.
I was gonna investigate but got delayed by other projects.
But for common usage (without artificially creating 100s of connections) this branch is fine.
I could tidy things up and prepare a PR. I will take your suggestions into account.

btw, regarding your 3rd point, we only query for UID when connection was not found in ebpf.
In such case we query for UID via netlink to see if UID == 0 (an in-kernel connection)

The reason why an in-kernel connection will not be caught by our ebpf is because our ebpf hooks only to tcp...connect() and udp...send() kernel functions. But these 2 functions are NOT the only ones that can be used to establish a connection. Some in-kernel connections may be established using more low-level functions. (if memory serves me, wireguard uses ip...tunnel() function or similar).
There's more research that can be done to hook up to other kernel functions to catch all connections.

[gustavo-iniguez-goya]

thank you @themighty1 ! I know that we'll miss some connections, but this is better than what we have right now. If we wait til it's perfect we'll never add it. We'll keep improving it over time.

On the other hand, we still rely on /proc to discover the PID's cmdline and path. All the other solutions I've analyzed (*snitch) do the same, so we all are in the same boat. Even ntop do this.
But I think that we can get more info from the kernel without relying on ProcFS.

I've modified the commands tcptracer-bpfcc/tcpconnect-bpfcc (bpfcc-tools package) to get the full path to the binary from a task_struct, and although I haven't managed to get it properly (without garbage), it's definitely possible:

      struct task_struct *task = (struct task_struct *)bpf_get_current_task();
      // this still needs CONFIG_PROC_FS=y
      // exe_file -> /proc/<PID>/exe symlink
      bpf_probe_read_kernel_str(&evt4.path, sizeof(evt4.path), &task->mm->exe_file->f_path.dentry->d_name.name);

The f_path.dentry struct points to the filename of the process, and f_path->d_parent to the parent directory of the file. So we should read all the d_parent structs:

      bpf_probe_read_kernel_str(&evt4.path1, sizeof(evt4.path1), &task->mm->exe_file->f_path.dentry->d_parent->d_name.name);
      bpf_probe_read_kernel_str(&evt4.path2, sizeof(evt4.path2), &task->mm->exe_file->f_path.dentry->d_parent->d_parent->d_name.name);
---
PID      COMM           IP  SRC                  DST                  PORT
17172  Socket Threa 4  192.168.1.109    52.84.66.184     443    
PATH: /|x0|usr /|8V|lib  }x0}firefox-esr
INODE: 1597455

There's more research that can be done to hook up to other kernel functions to catch all connections.

Definitely. Besides investigating all those corner cases, maybe we could also hook fork and exec to get the path of a process in a easy way, and later correlate it by PID.

Since kernel 5.11 there's also this framework:
https://lwn.net/Articles/698226/
https://mjmwired.net/kernel/Documentation/bpf/bpf_lsm.rst (architecture independant)
https://www.kernel.org/doc/html/latest/security/lsm.html#lsm-capabilities-module
https://www.kernel.org/doc/htmldocs/lsm/framework.html

[themighty1]

I looked at bpf_get_current_comm() and realized that it return only the exe name excluding the path, so, yes, parsing task_struct is what we'll need to get the full path.

[gustavo-iniguez-goya]

and found corner-cases when connections were not picked-up by ebpf.

by the way, talking about (dark) corner cases:
https://blog.cloudflare.com/conntrack-turns-a-blind-eye-to-dropped-syns/
https://blog.cloudflare.com/conntrack-tales-one-thousand-and-one-flows/

another dark corner?

/sys/kernel/debug/tracing # cat trace_pipe

   Socket Thread-17215   [002] .... 3576146.553217: tcp_v4_connect <-__inet_stream_connect
   Socket Thread-17215   [002] .... 3576146.553227: inet_sock_set_state: family=AF_INET protocol=IPPROTO_TCP sport=0 dport=80 saddr=192.168.1.109 daddr=23.206.172.34 saddrv6=::ffff:192.168.1.109 daddrv6=::ffff:23.206.172.34 oldstate=TCP_CLOSE newstate=TCP_SYN_SENT
  irq/32-iwlwifi-622     [000] .... 3576146.622844: inet_sock_set_state: family=AF_INET protocol=IPPROTO_TCP sport=42070 dport=80 saddr=192.168.1.109 daddr=23.206.172.34 saddrv6=::ffff:192.168.1.109 daddrv6=::ffff:23.206.172.34 oldstate=TCP_SYN_SENT newstate=TCP_ESTABLISHED

notice that between TCP_SYN_SENT and TCP_ESTABLISHED states, the reported PID is not 17215, it changes to 622:

# cat /proc/622/cmdline
# ls -l /proc/622/exe
ls: can't read symbolic link '/proc/622/exe': no such file or directory

and this is a connection from inside a container:

<idle>-0       [006] .Ns. 3576147.614660: inet_sock_set_state: family=AF_INET protocol=IPPROTO_TCP sport=54882 dport=50051 saddr=172.17.0.2 daddr=172.17.0.1 saddrv6=::ffff:172.17.0.2 daddrv6=::ffff:172.17.0.1 oldstate=TCP_SYN_SENT newstate=TCP_CLOSE

the process that opened that connection is opensnitchd running on docker that connects to a remote GUI, and usually it's reported fine:

opensnitchd-18187   [002] .... 3580748.631319: inet_sock_set_state: family=AF_INET protocol=IPPROTO_TCP sport=0 dport=50051 saddr=172.17.0.2 daddr=172.17.0.1 saddrv6=::ffff:172.17.0.2 daddrv6=::ffff:172.17.0.1 oldstate=TCP_CLOSE newstate=TCP_SYN_SENT

[gustavo-iniguez-goya]

So we should read all the d_parent structs:

https://github.com/aquasecurity/tracee/blob/main/tracee-ebpf/tracee/tracee.bpf.c#L1112

[gustavo-iniguez-goya]

thank you very much for this @themighty1 . I used to reproduce the "unknown connection" issue with maps, either with maps.google.com (firefox or chrome) or gnome-maps, zooming in and out continuously and now the problem is gone.

[gustavo-iniguez-goya]

regarding Linux Kernel connections, the ones I've seen are from chrome, but with wrong PID and UID and empty cmdline (because the PID is not found):

All connections to the same IP, so I'm going to remove the "Linux Kernel" -100 check, and see if we find it on already known connections.

Maybe we should just discard, or not add to the map those connections if PID + UID are invalid.

[themighty1]

@gustavo-iniguez-goya, you're welcome! Thank you for moving the project forward.
Im not sure I undestood your last comment because after checking ebpf map, we check already established and after that we check netlink before we decide that a connection is from the kernel.

[gustavo-iniguez-goya]

yes, even after all the checks, sometimes I got "kernel" connections, but since I stopped the docker daemon I haven't had these problems again. I'll try to reproduce it this week.

Regarding this log:

ebpf warning: eBPF packet with unknown source IP:

at least in my case, it's caused by connections traversing the box. Typically connections originated from a VM or a container.

IPv4: true getPidFromEbpf(), connection not found: tcp 80 34.107.221.82 192.168.1.102 40240
[2021-04-08 14:14:40]  DBG  netlink socket error: Warning, no message nor error from netlink - 80:34.107.221.82 -> 192.168.1.102:40240
[2021-04-08 14:14:40]  WAR  ebpf warning: eBPF packet with unknown source IP: tcp, 80:34.107.221.82 -> 192.168.1.102:40240

But all in all it's working great.

[gustavo-iniguez-goya]

module cross-compiled for i386 and arm successfully. On raspberry pi, the kernel lacks some BPF features (CONFIG_BPF_SYSCALL), so it fails with the error: error while loading map "maps/tcpMap": function not implemented

On the other hand, github.com/iovisor/gobpf is not packaged in Debian, so it breaks my build setup, but I've managed to configure in another way.

[themighty1]

great, well done!