workaround for hitting BPF max stack size

[williamtu]

I declare another "struct Headers hd__" at the deparser code block, so that the live range of "hd" is reset at deparser.

An example of using xdp7.p4, before the patch, we use 616 byte stack memory

# llvm-objdump -S -no-show-raw-insn xdp7.o | grep "r10 -" | awk '{print $7}' | sort -n
616
After the patch:
392

What do you think?

index db5fff7..433d546 100644
--- a/tests/xdp7.c
+++ b/root/xdp7.c
@@ -1,4 +1,4 @@
-/* Automatically generated by p4c-xdp from xdp7.p4 on Thu Mar  2 08:04:24 2017
+/* Automatically generated by p4c-xdp from xdp7.p4 on Wed Mar  1 09:52:18 2017
  */
 #include "xdp7.h"
 #define KBUILD_MODNAME "xdptest"
@@ -133,6 +133,7 @@ int ebpf_filter(struct xdp_md* skb){
             .ebpf_valid = 0
         },
     };
+    struct Headers hd__;
     unsigned ebpf_packetOffsetInBits = 0;
     enum ebpf_errorCodes ebpf_errorCode = NoError;
     void* ebpf_packetStart = ((void*)(long)skb->data);
@@ -379,12 +380,13 @@ int ebpf_filter(struct xdp_md* skb){
     }
     /* deparser */
     {
+        hd__ = hd;
         {
-            if (hd.ethernet.ebpf_valid) ebpf_outHeaderLength += 112;
-            if (hd.ipv4.ebpf_valid) ebpf_outHeaderLength += 160;
-            if (hd.icmp.ebpf_valid) ebpf_outHeaderLength += 32;
-            if (hd.udp.ebpf_valid) ebpf_outHeaderLength += 64;
-            if (hd.tcp.ebpf_valid) ebpf_outHeaderLength += 160;
+            if (hd__.ethernet.ebpf_valid) ebpf_outHeaderLength += 112;
+            if (hd__.ipv4.ebpf_valid) ebpf_outHeaderLength += 160;
+            if (hd__.icmp.ebpf_valid) ebpf_outHeaderLength += 32;
+            if (hd__.udp.ebpf_valid) ebpf_outHeaderLength += 64;
+            if (hd__.tcp.ebpf_valid) ebpf_outHeaderLength += 160;
         }
         bpf_xdp_adjust_head(skb, BYTES(ebpf_packetOffsetInBits) - BYTES(ebpf_outHeaderLength));
         ebpf_packetStart = ((void*)(long)skb->data);
@@ -392,230 +394,230 @@ int ebpf_filter(struct xdp_md* skb){
         ebpf_packetOffsetInBits = 0;
         u8 hit_0;
         {
-            /* packet.emit(hd.ethernet)*/
-            if (hd.ethernet.ebpf_valid) {
+            /* packet.emit(hd__.ethernet)*/
+            if (hd__.ethernet.ebpf_valid) {
                 if (ebpf_packetEnd < ebpf_packetStart + BYTES(ebpf_packetOffsetInBits + 112)) {
                     ebpf_errorCode = PacketTooShort;
                     return XDP_ABORTED;
                 }
-                ebpf_byte = ((char*)(&hd.ethernet.destination))[0];
+                ebpf_byte = ((char*)(&hd__.ethernet.destination))[0];
                 write_byte(ebpf_packetStart, BYTES(ebpf_packetOffsetInBits) + 0, (ebpf_byte) << 0);
-                ebpf_byte = ((char*)(&hd.ethernet.destination))[1];
+                ebpf_byte = ((char*)(&hd__.ethernet.destination))[1];
                 write_byte(ebpf_packetStart, BYTES(ebpf_packetOffsetInBits) + 1, (ebpf_byte) << 0);
-                ebpf_byte = ((char*)(&hd.ethernet.destination))[2];
+                ebpf_byte = ((char*)(&hd__.ethernet.destination))[2];
                 write_byte(ebpf_packetStart, BYTES(ebpf_packetOffsetInBits) + 2, (ebpf_byte) << 0);
-                ebpf_byte = ((char*)(&hd.ethernet.destination))[3];
+                ebpf_byte = ((char*)(&hd__.ethernet.destination))[3];
                 write_byte(ebpf_packetStart, BYTES(ebpf_packetOffsetInBits) + 3, (ebpf_byte) << 0);
-                ebpf_byte = ((char*)(&hd.ethernet.destination))[4];
+                ebpf_byte = ((char*)(&hd__.ethernet.destination))[4];
                 write_byte(ebpf_packetStart, BYTES(ebpf_packetOffsetInBits) + 4, (ebpf_byte) << 0);
-                ebpf_byte = ((char*)(&hd.ethernet.destination))[5];
+                ebpf_byte = ((char*)(&hd__.ethernet.destination))[5];
... <skip the rest>

[mihaibudiu]

How about declaring hd as volatile?

[williamtu]

then it uses only 136 byte
But volatile generates too large program, so verifier says:
"BPF program is too large. Proccessed 65537 insn"

[mihaibudiu]

That is weird, you would not expect this program to be so large.

Maybe you can play more with volatile; perhaps instead of a second structure hd__ you can just use a volatile pointer to hd and use to make a read between the parser and the deparser.

[williamtu]

OK.
Actually the program size (number of instructions in the program) with and without volatile is roughly the same. Then it's weird that BPF verifier fails.

[williamtu]

I actually like the generated with volatile. With volatile, the compile spills the ethernet src/dst to stack using 1 byte instead of 8 byte, so it uses much less stack. However, there is no reordering at all, it's easier to read, but maybe have some performance overhead?

At lease when reading out the hd.x.y, it will read it from the stack and with the right size (no longer fixed to 8 byte).
With volatile, the size is 1232 line of instructions
Without volatile, it is 1363.

Now I suspect there might be an issue in verifier.

[kwjjyn]

Hi,
I using p4c-xdp to compile xdp7.p4 to get .c and .h files. And if I just compile the xdp7.c using clang directly，there's an error that

invalid stack off=-600 size=8

But when I add volatile before struct Headers hd in the xdp7.c , it would be no problem to load.
If there are any other ways to generate .c file with volatile from .p4 file using p4c-xdp compiler?
Otherwise I must add volatile before hd for ervery .c files which cannot be loaded into the kernel due to the hitting BPF stack.
Could you give me some advices?

[osinstom]

@williamtu did you find the suitable solution for this problem ? We're also facing it in our uBPF backend. The volatile keyword helps to overcome BPF stack size problem, but the verifier doesn't allow to inject program then.