Support tracing XDP #339

Asphaltt · 2024-03-04T14:58:28Z

Support tracing XDP

Add an option --filter-trace-xdp to trace all XDP progs on host by
fentry-ing on the progs, like the way tracing tc-bpf
Asphaltt@2347755.

The diff from tracing tc-bpf:

Not support to filter mark.
No mark in meta output.
No proto in meta output.
Not support --output-skb.

Example with --filter-trace-xdp --filter-trace-tc:

./pwru --filter-trace-xdp --filter-trace-tc --output-limit-lines 10 --filter-func 'xxx' --output-meta --output-tuple icmp
2024/03/04 14:26:47 Listening for events..
               SKB    CPU          PROCESS                     FUNC
0xffffbb8cc062cc28      7     [<empty>(0)]               dummy(xdp) netns=4026531840 mark=0x0 iface=2(ens33) proto=0x0000 mtu=1500 len=98 192.168.241.1:0->192.168.241.128:0(icmp)
0xffff95ed4596aa00      7     [<empty>(0)]                dummy(tc) netns=4026531840 mark=0x0 iface=2(ens33) proto=0x0800 mtu=1500 len=98 192.168.241.1:0->192.168.241.128:0(icmp)
0xffffbb8cc062cc28      7     [<empty>(0)]               dummy(xdp) netns=4026531840 mark=0x0 iface=2(ens33) proto=0x0000 mtu=1500 len=98 192.168.241.1:0->192.168.241.128:0(icmp)
0xffff95ed4596ba00      7     [<empty>(0)]                dummy(tc) netns=4026531840 mark=0x0 iface=2(ens33) proto=0x0800 mtu=1500 len=98 192.168.241.1:0->192.168.241.128:0(icmp)
0xffffbb8cc062cc28      7     [<empty>(0)]               dummy(xdp) netns=4026531840 mark=0x0 iface=2(ens33) proto=0x0000 mtu=1500 len=98 192.168.241.1:0->192.168.241.128:0(icmp)
0xffff95ed4596a700      7     [<empty>(0)]                dummy(tc) netns=4026531840 mark=0x0 iface=2(ens33) proto=0x0800 mtu=1500 len=98 192.168.241.1:0->192.168.241.128:0(icmp)
0xffffbb8cc062cc28      7     [<empty>(0)]               dummy(xdp) netns=4026531840 mark=0x0 iface=2(ens33) proto=0x0000 mtu=1500 len=98 192.168.241.1:0->192.168.241.128:0(icmp)
0xffff95ed4596b500      7     [<empty>(0)]                dummy(tc) netns=4026531840 mark=0x0 iface=2(ens33) proto=0x0800 mtu=1500 len=98 192.168.241.1:0->192.168.241.128:0(icmp)
0xffffbb8cc062cc28      7     [<empty>(0)]               dummy(xdp) netns=4026531840 mark=0x0 iface=2(ens33) proto=0x0000 mtu=1500 len=98 192.168.241.1:0->192.168.241.128:0(icmp)
0xffff95ed4596b400      7     [<empty>(0)]                dummy(tc) netns=4026531840 mark=0x0 iface=2(ens33) proto=0x0800 mtu=1500 len=98 192.168.241.1:0->192.168.241.128:0(icmp)
2024/03/04 14:26:52 Printed 10 events, exiting program..

jschwinger233 · 2024-03-05T06:58:33Z

Thanks! I found it may be a technical obstacle for us to trace a frame's lifetime, it seems no way to link an xdp_buff to an skb: the "skb addr" is different, we can't know which skb an xdp_buff has turned to.

Asphaltt · 2024-03-05T08:25:06Z

There is a possible approach to link an xdp_buff and an sk_buff: xdp_buff->data_hard_start and sk_buff->head.

I'll do a POC to confirm it later.

Asphaltt · 2024-03-05T12:37:01Z

It seems great to use xdp_buff->data_hard_start and sk_buff->head.

root@leon-vm ~/P/l/pwru (feature/xdp)# ./pwru --filter-trace-xdp --filter-trace-tc --output-limit-lines 10 --filter-func 'xxx' --output-meta --output-tuple icmp
2024/03/05 12:31:01 Attaching kprobes (via kprobe-multi)...
0 [_________________________________________________________________________________________________________________________________________________] ?% ? p/s
2024/03/05 12:31:01 Attached (ignored 0)
2024/03/05 12:31:01 Listening for events..
               SKB    CPU          PROCESS                     FUNC
0xffff95ee12521000      7     [<empty>(0)]               dummy(xdp) netns=4026531840 mark=0x0 iface=2(ens33) proto=0x0000 mtu=1500 len=98 192.168.241.1:0->192.168.241.128:0(icmp)
0xffff95ee12521000      7     [<empty>(0)]                dummy(tc) netns=4026531840 mark=0x0 iface=2(ens33) proto=0x0800 mtu=1500 len=98 192.168.241.1:0->192.168.241.128:0(icmp)
0xffff95ee12521000      7     [<empty>(0)]               dummy(xdp) netns=4026531840 mark=0x0 iface=2(ens33) proto=0x0000 mtu=1500 len=98 192.168.241.1:0->192.168.241.128:0(icmp)
0xffff95ee12521000      7     [<empty>(0)]                dummy(tc) netns=4026531840 mark=0x0 iface=2(ens33) proto=0x0800 mtu=1500 len=98 192.168.241.1:0->192.168.241.128:0(icmp)
0xffff95ee12521000      7     [<empty>(0)]               dummy(xdp) netns=4026531840 mark=0x0 iface=2(ens33) proto=0x0000 mtu=1500 len=98 192.168.241.1:0->192.168.241.128:0(icmp)
0xffff95ee12521000      7     [<empty>(0)]                dummy(tc) netns=4026531840 mark=0x0 iface=2(ens33) proto=0x0800 mtu=1500 len=98 192.168.241.1:0->192.168.241.128:0(icmp)
0xffff95ee12521000      7     [<empty>(0)]               dummy(xdp) netns=4026531840 mark=0x0 iface=2(ens33) proto=0x0000 mtu=1500 len=98 192.168.241.1:0->192.168.241.128:0(icmp)
0xffff95ee12521000      7     [<empty>(0)]                dummy(tc) netns=4026531840 mark=0x0 iface=2(ens33) proto=0x0800 mtu=1500 len=98 192.168.241.1:0->192.168.241.128:0(icmp)
0xffff95ee12521000      7     [<empty>(0)]               dummy(xdp) netns=4026531840 mark=0x0 iface=2(ens33) proto=0x0000 mtu=1500 len=98 192.168.241.1:0->192.168.241.128:0(icmp)
0xffff95ee12521000      7     [<empty>(0)]                dummy(tc) netns=4026531840 mark=0x0 iface=2(ens33) proto=0x0800 mtu=1500 len=98 192.168.241.1:0->192.168.241.128:0(icmp)
2024/03/05 12:31:03 Printed 10 events, exiting program..
2024/03/05 12:31:03 Detaching kprobes...
0 [_________________________________________________________________________________________________________________________________________________] ?% ? p/s
root@leon-vm ~/P/l/pwru (feature/xdp)# git diff
diff --git a/bpf/kprobe_pwru.c b/bpf/kprobe_pwru.c
index f60913c..7411d8a 100644
--- a/bpf/kprobe_pwru.c
+++ b/bpf/kprobe_pwru.c
@@ -391,7 +391,7 @@ int BPF_PROG(fentry_tc, struct sk_buff *skb) {
        if (!handle_everything(skb, ctx, &event))
                return BPF_OK;

-       event.skb_addr = (u64) skb;
+       event.skb_addr = (u64) BPF_CORE_READ(skb, head);
        event.addr = BPF_PROG_ADDR;
        event.type = EVENT_TYPE_TC;
        bpf_map_push_elem(&events, &event, BPF_EXIST);
@@ -488,7 +488,7 @@ int BPF_PROG(fentry_xdp, struct xdp_buff *xdp) {
        event.pid = bpf_get_current_pid_tgid() >> 32;
        event.ts = bpf_ktime_get_ns();
        event.cpu_id = bpf_get_smp_processor_id();
-       event.skb_addr = (u64) xdp;
+       event.skb_addr = (u64) BPF_CORE_READ(xdp, data_hard_start);
        event.addr = BPF_PROG_ADDR;
        event.type = EVENT_TYPE_XDP;
        bpf_map_push_elem(&events, &event, BPF_EXIST);

Let me apply this diff.

brb · 2024-03-17T16:02:16Z

Thanks! I've just tried running, and got:

2024/03/17 16:59:32 failed to trace XDP progs: failed to trace bpf progs: failed to get entry function name: bpf program xdp_foo does not have BTF

Maybe we could ignore such progs instead of panic?

brb · 2024-03-22T08:45:43Z

It seems great to use xdp_buff->data_hard_start and sk_buff->head.

Are you saying that both point to the same in the case XDP moves to the host (e.g., TC)?

OT: could you drop the last commit, and just do git rebase origin/main?

Asphaltt · 2024-03-22T09:23:14Z

Are you saying that both point to the same in the case XDP moves to the host (e.g., TC)?

Yes. As the output in previous reply.

When to build xdp_buff from data: xdp_prepare_buff
When to build skb from data: __finalize_skb_around

OT: could you drop the last commit, and just do git rebase origin/main?

Oh, sorry for my bad "merge". I will rebase it later.

brb · 2024-04-05T15:15:32Z

When to build xdp_buff from data: xdp_prepare_buff
When to build skb from data: __finalize_skb_around

With these changes --trace-skb starts tracking some unrelated packets (see at the very end in after.log). I've traced NodePort BPF traffic on Kind with ./pwru --filter-track-skb --filter-trace-tc --output-meta --output-tuple 'port 30828'.

after.log - with PR changes
before.log - before PR changes

Asphaltt · 2024-04-06T03:20:14Z

With these changes --trace-skb starts tracking some unrelated packets (see at the very end in after.log).

It seems it does not remove the skb addr from skb_addresses bpf map.

Did it log Warn: kfree_skbmem not found, pwru is likely to mismatch skb due to lack of skb lifetime management when to run with --filter-track-skb? If it logged it, it would track the cloned/copied skbs without removing them from skb_addresses bpf map.

I'd like to fix this code logic bug in #340.

brb · 2024-04-06T05:40:16Z

Did it log Warn: kfree_skbmem not found, pwru is likely to mismatch skb due to lack of skb lifetime management when to run with --filter-track-skb? If it logged it, it would track the cloned/copied skbs without removing them from skb_addresses bpf map.

Noup, it didn't log. Shall I give it a go #340?

Asphaltt · 2024-04-06T07:33:15Z

Shall I give it a go #340?

Sure.

brb · 2024-04-08T12:28:20Z

Indeed, adding #340 to this branch resolved the last traces hiccup. Thanks! However, I still miss some traces when running this PR with #340. E.g., tcp_v4_conn_request (before.txt, after.txt). Any idea regarding this?

In bpf verifier, it denies that FENTRY/FEXIT cannot trace BTF-not-annotated bpf progs. Or, an error "FENTRY/FEXIT program can only be attached to another program annotated with BTF" occurs. Signed-off-by: Leon Hwang <hffilwlqm@gmail.com>

It's to trace XDP progs in future commits. Signed-off-by: Leon Hwang <hffilwlqm@gmail.com>

Add an option --filter-trace-xdp to trace all XDP progs on host by fentry-ing on the progs, like the way of tracing tc-bpf 2347755. The diff from tracing tc-bpf: 1. Not support to filter mark. 2. No mark in meta output. 3. No proto in meta output. 4. Not support --output-skb. Signed-off-by: Leon Hwang <hffilwlqm@gmail.com>

As we know, the hard start of packet data is same between `struct sk_buff` and `struct xdp_buff`. So, we can track skb by it, and output it as skb address. Signed-off-by: Leon Hwang <hffilwlqm@gmail.com>

Asphaltt requested a review from a team as a code owner March 4, 2024 14:58

Asphaltt requested review from brb and removed request for a team March 4, 2024 14:58

Asphaltt force-pushed the feature/xdp branch from 3ad9134 to 8bbc13b Compare March 4, 2024 15:05

Asphaltt force-pushed the feature/xdp branch 2 times, most recently from 5ec35e9 to 7b97571 Compare March 10, 2024 03:56

This was referenced Mar 17, 2024

Ability to monitor XDP handled packets cilium/hubble#1339

Open

datapath: Emit trace events in XDP for debug purpose cilium/cilium#19703

Open

Asphaltt force-pushed the feature/xdp branch from 2bcc4fa to f0fe6ed Compare March 22, 2024 12:21

Asphaltt added 4 commits April 25, 2024 22:38

Skip trace BTF-not-annotated bpf progs

ad27ad5

In bpf verifier, it denies that FENTRY/FEXIT cannot trace BTF-not-annotated bpf progs. Or, an error "FENTRY/FEXIT program can only be attached to another program annotated with BTF" occurs. Signed-off-by: Leon Hwang <hffilwlqm@gmail.com>

bpf: Add bpf_endian.h

8b8274d

It's to trace XDP progs in future commits. Signed-off-by: Leon Hwang <hffilwlqm@gmail.com>

Track and output hard start of packet data

97d6611

As we know, the hard start of packet data is same between `struct sk_buff` and `struct xdp_buff`. So, we can track skb by it, and output it as skb address. Signed-off-by: Leon Hwang <hffilwlqm@gmail.com>

Asphaltt force-pushed the feature/xdp branch from f0fe6ed to 97d6611 Compare April 25, 2024 15:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support tracing XDP #339

Support tracing XDP #339

Asphaltt commented Mar 4, 2024

jschwinger233 commented Mar 5, 2024

Asphaltt commented Mar 5, 2024

Asphaltt commented Mar 5, 2024

brb commented Mar 17, 2024 •

edited

brb commented Mar 22, 2024 •

edited

Asphaltt commented Mar 22, 2024

brb commented Apr 5, 2024

Asphaltt commented Apr 6, 2024

brb commented Apr 6, 2024

Asphaltt commented Apr 6, 2024

brb commented Apr 8, 2024

Support tracing XDP #339

Are you sure you want to change the base?

Support tracing XDP #339

Conversation

Asphaltt commented Mar 4, 2024

jschwinger233 commented Mar 5, 2024

Asphaltt commented Mar 5, 2024

Asphaltt commented Mar 5, 2024

brb commented Mar 17, 2024 • edited

brb commented Mar 22, 2024 • edited

Asphaltt commented Mar 22, 2024

brb commented Apr 5, 2024

Asphaltt commented Apr 6, 2024

brb commented Apr 6, 2024

Asphaltt commented Apr 6, 2024

brb commented Apr 8, 2024

brb commented Mar 17, 2024 •

edited

brb commented Mar 22, 2024 •

edited