VPP vs NFF Go: Productivity Comparison
Areg Melik-Adamyan edited this page Aug 7, 2019
·
12 revisions
Below are live notes from our summer intern Aditya Gadiyar, who was not familiar with network programming or low-level system programming and was asked to perform a simple task: decrease TTL of a packet and drop by some condition. Our goal was to understand what problems non-ninja programmers will encounter during this task and how much productivity gain NFF-Go has over VPP.
- Read NFF-GO documentation
- Read NFF-GO documentation
- Install Go on all machines
- Clone and build NFF-Go
- Problems with cache and temp directories while building
- Bind network ports to Kernel modules
- Task: receive packets at port 0 and send to port 1
- Problems with Go language: imports, return multiple types, nils
- Task: receive packets at port 0, drop half, send to port 1
- Spent time understanding when it is more suitable to use SetHandlerDrop vs. SetPartitioner
- Task: Generate packets at Machine 4, half with TTL = 1, the other half with TTL = 2. Receive at Machine 5, decrement TTL, drop if required. Send the other half to Machine 6.
- Problem: The difference between SetGenerator and SetFastGenerator is partially clear. It is clear that SetFastGenerator allows the programmer to choose the target speed of packet flow. However, “SetFastGenerator returned uint64 channel which can be used to change current target speed” is less clear. Had to examine flow.go to understand that target speed can be dynamically changed.
- Problem: Had trouble identifying syntax error with parameter
c *flow.Contextused in UDFs throughout the documentation. Had to go through examples and flow.go to realize that it must bec flow.UserContext. The purpose of the Context parameter is also slightly vague in the documentation. It is only defined as a “developer-defined interface that is used for some global environment.” In the majority of examples, and throughout classes like flow.go, this parameter is just nil. For my purposes, I didn’t have to use the Context parameter. With decrementTTL function – just like for packet generation – it wasn’t clear that context parameter for UDFs is ‘c flow.UserContext’ - Problem: When initializing current packet from SetGenerator into an empty IPv4 packet, I understood that the function packet.InitEmptyIPv4 Packet adds pointers to IPv4 and Ethernet headers and then pre-allocates additional bytes for data payload. Since I was not sending packets with any data (beyond TTL in the header), I initialized this pre-allocated size to 0 bytes. The documentation does not make it clear that the Ethernet frame has a minimum size of 64 bytes. So in addition to 14 bytes for Ethernet header and 20 bytes for IPv4 header, 30 additional bytes had to be initialized to 0 so the packets were defined. It was unclear that I, therefore, had to pass 30 as a parameter to
packet.InitEmptyIPv4Packet() - Problem: Documentation does not clearly explain that
flow.SystemStop()only needs to be called to stop a thread when multiple threads are being run in parallel. I’d initially includedflow.SystemStop()at the end of the ‘main’ function, but then realized there was no need for this since my program was a single thread.
- Code cleanup for commit, committing
- Read VPP documentation to gain a better understanding of its basic structure and functions
- Reviewed C and C++ programming
- Read “An Excursion in VPP” from Tsyesika’s Blog.
- Started going through VPP’s tutorials.
- Downloaded, installed, built, and ran VPP.
- Problem: There are around 4 websites/methods through which VPP/FDio can be installed from binary packages. Each method installs a slightly different version of VPP. The key differences between these versions is not clearly explained in VPP’s documentation. Ultimately, I cloned the git repository from gerrit.fd.io/r/vpp
- Problem: Building VPP for the first time resulted in several warnings and a few errors, including an error in the Makefile. While the source of these errors was unclear, correcting the Makefile allowed for a successful build.
- Bound network ports to kernel modules.
- Continued to read through VPP documentation
- NFF-Go’s documentation lists functions, provides an explanation of each function (including inputs/outputs/examples), and then a link to the source code.
- Conversely, VPP’s documentation solely provides source code with little explanation. As such, for someone new to network programming, it is much more challenging to start programming with VPP.
- Completed VPP tutorials on forwarding, dropping, and various other basic configurations.
- Task #1: Receive packets generated by machine-4 at machine-5. Forward these packets to machine-6. DO NOT create plugin. Complete task solely by configuring VPP.
- Learned how to use VPP terminal in order to interact with graph nodes.
- Learned how to run VPP in debug mode to gain a better understanding of forwarding errors.
- Problem: When using ‘sudo make run’ to run VPP, the process often hangs when setting up PCI ports, particularly when the network device is active (eno1 and eno2). This required me to exit the terminal and reconnect. This issue has continued to often occur throughout my time using VPP, and I’m yet to diagnose the source of the problem. Later learned that this was due to running multiple instances of VPP unknowingly. DPDK is only meant to handle one instance at a time.
- Task #1 (Continued):
- Learned about feature arcs and how to customize the ordering of nodes.
- Learned about creating an IP-interface within VPP for forwarding.
- Essentially an IP link within VPP similar to making an IP link in linux.
- Learned how to use ‘next_node,’ ‘runs_after,’ and ‘runs_before’ functions to configure VPP.
- Sent packets to verify that configuration was correct.
- Used my own packet generator written in NFF-Go to send packets from machine-4 to machine-5. Successfully forwarded packets to machine-6.
- Task #2: Generate IPv4 packets at Machine 4, first half with TTL = 1, second half with TTL = 2. Receive at Machine 5 and create a VPP plugin that decrements TTL, drops packets if required. Route the other half of packets to Machine 6.
- Examined source code from files such as ip4_forwarding.c and ip4_punt_or_drop.c to understand the logic behind dropping packets, as written in C.
- NFF-Go’s functions for developers are relatively simple and call much more complex functions that are in the source code.
- VPP doesn’t have simple functions that call various complex pre-written functions. Therefore, functions have to largely be written from scratch.
- These points – combined with the fact that it is generally simpler to program with GoLang rather than C – make VPP more challenging to program for someone new to network programming.
- Gained a better understanding of how buffers and vectors are used throughout VPP for packet manipulation.
- Task #2 (Continued):
- Ran make-plugin.sh script that automatically creates the source files necessary to build a VPP plugin.
- Learned about the purpose of each of the plugin’s source files, particularly ‘vppDecrementTTL.c’ and ‘node.c.’
- Examined the code in files generated for plugin to gain better understanding of dual-while-loop structure.
- Continued to reference documentation to learn about data types from ‘vlib,’ ‘vnet,’etc.
- Learned how to use ‘next_node, runs_after, and runs_before’ functions to hook my plugin into the graph.
- Rebuilt VPP and sent packets to ensure that my node was correctly connected within ‘device-input’ feature arc.
- Studied source code for “ip4-drop” node to understand how to decrement TTL and drop packets using VPP functions in C.
- Completed packet-processing code for plugin, including additions to main C file and node.c.
- Problem: Configuration of DPDK ports had changed overnight. Tried to run DPDK-devbind.py script to correctly bind ports, but for some reason, the process would hang. This required me to exit the terminal and reconnect every time I attempted to bind ports. Spent several hours trying to debug this issue.
- Later learned that this was due to running multiple instances of VPP unknowingly. DPDK is only meant to handle one instance at a time.
- Problem with ports from previous day persisted. Reached out to DPDK engineer for help with this issue.
- He had me run several tests to diagnose the issue and gave me the following steps to follow:
- We determined that I had several DPDK-devbind.py tasks running in the background that shouldn’t have been there. Deleted each of these tasks.
- Loaded ‘uio’, ran ‘sudo modprobe uio’, and ran ‘sudo insmod (RTE_SDK)//kmod/igb_uio’
- Rebuilt VPP
- Restarted lab machine-5 in order for changes to take effect
- Problem: Upon restarting lab machine-5 as per engineer’s guidance, I lost access to the machine and wasn’t able to ssh via Putty.
- Problem with machine-5 connectivity from previous day persisted. Spent several hours trying to debug this issue.
- Reached out to tech-support for help. They helped restore connectivity to machine-5.
- Followed engineer’s steps again to debug and set up ports. Successfully bound network ports to DPDK.
- Used pktgen to verify correct port configuration.
- Rebuilt VPP such that my plugin was loaded as part of VPP’s graph.
- Problem: packets were being sent to machine-5. However, VPP was not routing any packets to machine-6.
- Determined that node was incorrectly hooked into VPP’s graph.
- Learned more about feature arcs, features, nodes, and how to hook a plugin into VPP’s graph correctly.
- VPP’s documentation implies that filling in the ‘runs_before’ and ‘runs_after’ fields will allow you to insert your feature into a feature arc. While this is technically true, it does not guarantee that packets will be directed through your feature within the overall graph. Without the ability to change the ‘runs_before’ and ‘runs_after’ fields for other build-in nodes, it is not possible to programmatically direct packets through your feature
- Later learned that custom features can be added to interfaces via the VPP terminal in order to direct packets through them
- Worked with VPP team in Chandler to learn more about network stacks, OSI, particularly L2 and L3.
- Learned more about DPDK’s structure and benefits (kernel space vs. user space, polling, sockets, virtual ports, DDIO, etc.)
- Learned about the functionality that VPP adds on top of DPDK.
- Worked with performance engineering team in Chandler lab.
- Learned how VPP is used for performance benchmarking for finding hardware bottlenecks
- Learned how to send trace packets to determine the ordering of nodes in VPP graph.
- Learned to use ‘vppctl show features verbose’ to see how each feature is associated with each arc
- More issues with VPP hanging every time I try ‘sudo make run’
- Problem: pktgen tool shows that packets are being received on machine-5 but VPP interfaces show that all packets are resulting in an ‘rx-miss’.
- Researched VPP and packet reception and learned that VPP requires MAC addresses to match, whereas DPDK pktgen likely doesn’t consider MAC addresses
- Later learned that this was due to running VPP and pktgen at the same time on the same machine. DPDK is only meant to handle a single instance.
- Many of VPP’s tutorials actually create multiple instances of VPP. These tutorials do not explain that doing so eliminates the ability to utilize DPDK.
- Learned that ‘promiscuous mode’ can be used to accept all packets without dropping
- Used vppctl to try and enable promiscuous mode for active interfaces. This did not rectify rx-miss issue.
- Tried to explore if DPDK pktgen can be configured to also match MAC addresses
- Realized that my plugin was hooked into the graph in an invalid way. Learned to use ‘vppctl show features verbose’ to see how each feature is associated with each arc.
- Had previously positioned my feature between two nodes that were on different arcs.
- Problem: running VPP on machine 5 was not turning interfaces up on DPDK pktgen on machine 4
- Problem: I previously thought that VPP would inexplicably freeze when I tried to run it. Since VPP runs on top of DPDK, only one VPP instance can be run at a time. In the past, I hadn’t fully quit each iteration of VPP. This led to multiple iterations being run in parallel, resulting in VPP failing to run properly.
-
killall -9didn’t work - Wasn’t able to end all these VPP tasks. VPP seems to have an issue where multiple instances prevent you from deleting any of these instances.
- Tried to reboot machine-5. Machine lost connectivity and I lost access.
- Tried to reboot machine 5
- However, VPP parallel instances were using a lot of memory and the system did not shut down because project directory was busy and couldn’t be unmounted.
- Therefore, the machine lost connectivity
- Emailed tech support: System had to be power-cycled and was brought back online again.
- After reboot, machine-5 had no instances of VPP running.
- Since then, I’ve made sure to quit every instance of VPP/pktgen to avoid parallel instances. Since realizing this, freezing has not occurred again.
- Tried to debug rx-miss
- Set DST MAC on port 1 of machine 4 to MAC address of port 0 on machine 5.
- With the proper MAC address, VPP was able to correctly receive packets without any rx-miss issues
- Also learned how to set virtual interfaces in VPP to promiscuous mode so that MAC addresses aren’t necessary.
- Problem: Although VPP was loading my plugin and although it was correctly implemented in a feature arc, packets weren’t necessarily directed to it.
- Ran packet trace and saw that my plugin was being bypassed every time.
- This makes sense: defining ‘runs_before’ and ‘runs_after’ adds new vertices, but doesn’t remove previous vertices. So if packets previously met some conditions and accordingly travelled on a specific path, it will not change paths when a new path (to my feature) is added.
- Learned how to add a specific feature to an interface. Added my vppDecTTL plugin to FortyGigabitEthernet6/0/0 on device-input arc.
- Another packet trace showed that packets (from pktgen) were successfully received by my feature when interfaces were put into promiscuous mode.
- Now that my feature is implemented in the VPP graph, I have to try and forward packets from machine 4 to machine 6.
- Problem: Tried to set next node to “interface-tx”. However, without the right MAC address, it seems that VPP sends packets to some default arbitrary MAC address
- The MAC address to which VPP was outputting packets did not belong to machine4/5/6.
- The same result occurred when packets were sent to “interface-output.”
- Next step: Try to update the src/dst MAC addresses of packets received at machine 5 before routing them to machine 6
- Problem: While DPDK does not require MAC addresses in packets to match, VPP does require valid MAC addresses for receiving and sending
- First tried to modify built-in MAC-SWAP functionality from sample plugin to change MAC addresses. This changed the MAC addresses for trace functions, but not the MAC addresses of the functions themselves.
- VPP documentation does not explicitly outline this behavior, nor the purpose of only changing values in trace data structures.
- Then wrote a local helper function to insert custom hardcoded MAC addresses in packets.
- While MAC addresses were now correctly updated, packets were still not being sent to machine 6.
- Worked on trying to prove that packet-dropping logic works correctly
- Manually initialized hardcoded TTL values and ran a packet trace to confirm that the rest of the plugin was correctly implemented.
- In order to send packets to machine 6, packets have to be “transferred” from port 0 on machine 5 to port 1 on machine 5. Need to understand how VPP sends packets between interfaces.
- Learned more about how NFFGO does this in order to see if there are similarities with VPP
- Buffers -> rings -> flow
- VPP doesn’t use its own version of rings, but it does store packets in buffers.
- Reached out to VPP engineers to understand how to link interfaces. Suggestions:
- ‘Show l2 fib’ shows any ip connections between interfaces (my configuration had no connections set up)
- ‘set int l2 xconnect’ to establish a cross-connection between interfaces to test the configuration between interfaces. This confirmed that packets could be sent between port 0/1 on VPP.
- ‘set int l2 bridge’ to add interfaces to bridge domain
- These methods partially worked because they sent all packets received by one interface to the other interface. However, in doing so, they also bypassed my ‘vppDecTTL’ feature and were sent directly from ‘ethernet-input’ to ‘interface-output’.
- VPP documentation does not explain how to direct packets on a bridge domain towards certain feature arcs.
- Hypothesis: since all details in l2 are accurate, but an l2 link does not correctly forward packets, an L3 link of some sort must be necessary.
- Explored how to set up an IP link via VPP.
- Was successfully able to set up an IP link via Linux. This allowed packets to be correctly forwarded to machine 6. Once again, however, packets were bypassing my plugin.
- VPP has several commands in order to set up an IP link directly with VPP virtual interfaces. Having followed these commands, ‘show ip fib’ demonstrated that I had successfully established an IP link between interfaces; however, packets were not being sent from port 0 to port 1 on machine 5.
- VPP documentation did not provide a lot of explanations beyond its basic IP link tutorial. It can be inferred that the VPP IP link did not work because of certain setup characteristics specific to the lab machines.
- Pivoted away from forwarding packets to machine 6 to focus on running a working plugin on machine 5 to send packets back to machine 4.
- Problem: VPP only receives packets sent from pktgen. VPP crashes whenever it receives packets sent by my PacketGen program (written with NFFGO).
- There is something fundamentally different about the packets that are being generated by PacketGen and pktgen.
- Try to make PacketGen packets identical to those generated by pktgen.
- Wrote a simple NFFGO program on machine 5 that receives packets from machine 4 and dumps the raw bytes (hex dumper function)
- Worked on matching every element of IP header in PacketGen to pktgen: chksum, TTL, PacketID, packet length, src/dst IP
- Learned about endianness and bit/byte/nib/hex calcs in order to set custom values and make headers match.
- Problem: even after IP headers were identical, VPP crashed when receiving from PacketGen, yet received packets normally from pktgen.
- The packets only differed in their Ethernet headers;
- packet.initEmptyIPv4Packet() initializes MAC addresses to 00:00:00:00:00:00. This causes VPP to crash.
- After initializing MAC addresses with both real and dummy MAC addresses (just can’t be all zeroes), VPP received packets from PacketGen.
- VPP documentation implies that when VPP interfaces are in promiscuous mode, MAC addresses shouldn’t matter. However, this is clearly not the case.
- Problem: VPP is receiving packets from PacketGen but not reading the correct TTL value. Printing TTL shows that is being read as 254 every time, from both PacketGen and pktgen.
- Looked through VPP source code and examples to try and determine how IPv4 header is read. Even when I copy and paste examples and run them, the header is not read correctly.
- Experimented with various functions meant for accessing packet data.
- ‘Vlib_buffer_get_current’ returns type ‘data’ (packet data), even when return type is implicitly ‘ip4_header_t’.
- VPP documentation does not state that the return type of ‘vlib_buffer_get_current’ is implicit (later found to be based on a byte index pointer).
- Noticed that IP header was being read before Ethernet header. Hypothesized that ip->ttl (ttl var in IP header struct) was pointing to a byte position in the Ethernet header instead.
- Learned about how C uses offsets, pointers, and memory references.
- Compared byte length of Ethernet and IP header to approximately find the byte in the Ethernet header with an offset that corresponds to TTL in IP header.
- Found that TTL was corresponding to the third byte of the destination MAC address in the Ethernet header.
- Verified this by changing the third byte of the MAC address from ‘fe’ to ‘01’ and ‘02’ (the desired TTL values) and saw that the mistaken TTL was correctly displayed.
- Packets are sent in buffers and individual packets are accessed via buffer indices. This buffer index (generally initialized throughout VPP as bi0 = from[0]) is used to access specific packets. In order to properly access TTL, this buffer index has to start at the beginning of the IP header
- Buffer index was previously pointing to the beginning of the packet (start of Ethernet header), meaning that the index was accessing a MAC address byte rather than the TTL byte.
- Used NFF-GO hexdump to dump packet byte contents and compare offsets. Determined that the buffer index had to be incremented by 14 in order to point to the beginning of the ip packet.
- Created a local pointer variable that was initialized to point to the default buffer index. Added an offset of 14 to this local index.
- Illustration of this:
[start of eth header] [start of IP header]
SRC MAC DST MAC LEN ID TTL HC SRC IP DST IP
3cfdfea54d90 3cfdfea54d61 0800 4500 002e 74f6 0000 01 06 c281 c0a80101 c0a80001
^ ^ ^
A C B
[start of IP header] TTL
4500002e74f6 000001 06 c281 c0a80101 c0a80001
- A: The default initial buffer index pointer
- B: TTL is the 9th byte of the IPv4 header
- C: Starting at the initial buffer index, the 9th byte is ‘fe’, which is the 3rd byte of the destination MAC address. The decimal form of ‘fe’ is 254, thus resulting in an apparent TTL of 254 for all packets received.
- The offset from ‘fe’ to ‘01’ (TTL) is 14 bytes (this makes sense because the length of the ethernet header is 14 bytes). Therefore, by incrementing the buffer index by 14, TTL can be correctly accessed.
- VPP documentation does not explicitly state that the buffer index pointer used in vlib_get_current (func to get packet data) always starts at the beginning of the packet, even when the expected return type is IPv4_header_t
- NFF-Go’s documentation contains generalized functions that call more specific functions in source code based on parameters inputted by the developer. These generalized functions are relatively simple to use because they are well-commented and have explanations/examples.
- NFF-Go is inherently easier to write because GoLang is simpler to write than C.
- NFF-Go’s simpler high-level functions and helpful documentation make it relatively straightforward for inexperienced developers to pick up network programming.
- NFF-Go enables developers to efficiently build performant network stacks and is also great for quickly prototyping new ideas.
- VPP’s documentation typically provides a link to the source code (no generalized functions). Thus, to understand how a VPP function works, a developer has to spend time understanding the code. Since the code is not thoroughly commented, and since variable/function names often consist of complex acronyms/ambiguous terms, it can be hard to understand this code for someone who is new to VPP.
- VPP’s documentation includes some helpful tutorials on how to perform basic tasks like running the VPP terminal, performing a packet trace, or setting up different configurations; this is very good for VPP beginners.
- VPP’s ability to write custom low-level C functions enables developers to have more control and customizability. The ability to tweak every detail of how packet flows are managed is best for maximizing performance.