Production use - and support

[mikebromwich]

I'm curious how whether production use of this library is widespread. We've recently ported one of our platforms to use nff-go, but have run into a couple of issues which are preventing us from having the confidence to move into production. I've raised one as a github issue - but there doesn't seem to be much activity and there has been no response for a couple of weeks. I've also tried E-Mailing the Intel team - but again, no response.

I appreciate that this is the nature of using open source libraries - and we are dependent on volunteers etc. helping out. However, we'd be prepared to pay for commercial support if it was an option. Without such support, it would seem to be a risky proposition for production use.

Any suggestions/comments appreciated!

Mike

[leoluk]

When we reached out to Intel a few years ago, they were indicating that Intel is using nff-go for customer projects. There are some general limitations (garbage collector-induced jitter, CGO overhead, no fast concurrent Cuckoo hashmap in Go) that are unlikely to go away but other than that, the project is solid.

CGO overhead may improve a lot in the future with https://go.googlesource.com/gollvm

You may want to email Areg (his email address is in the README) if you don't get a reply here.

Curious what problems did you face?

[leoluk]

nff-go has a rather rigid framework - it needs to keep the amount of CGO calls to a minimum, which is why it implements its own mbuf and so on, and has dedicated workers for rx/tx.

[mikebromwich]

Our issue seems to be that nff_go_send (in low.h) calls rte_eth_tx_burst(...), and directSend (also in low.h) also calls rte_eth_tx_burst(...), but there is nothing to stop them both doing so at the same time on the same queue from different threads - thus causing the panic/crash.

So, if packets are passing through the flow, and at the same time we need to respond to an ARP or ICMP request, this pretty reliably causes a crash. This seems pretty fundamental - so perhaps we have misunderstood and/or there is a simple solution.

[leoluk]

Yeah, direct use of DPDK primitives is not well-supported. You likely don't want to use directSend anyway, since it sends individidual packets and incurs the CGO call overhead on every invocation. It's better to use a custom generator in the flow model, since that'll batch it up and do as few CGO calls as possible.

If you need low-level DPDK primitives, Rust and https://github.com/capsule-rs/capsule may be a better choice. nff-go is more of a rapid prototyping tool (at least to us).

[mikebromwich]

Thanks - that's what we've done, but it's tied up another 2 cores for the occasional ARP reply or PING. Here's the flow graph we've ended up with (sorry about the poor art! It's from a code comment)

	// Access Side                                Core Side
	//    _____         _____         _____         _____
	//   |Recv |       |Handl|-o|    |Merge|       |Send |
	//   |  R0 |>----->|  H  |------>|  M  |>----->|  S1 |
	//   |_____|       |_____|       |_____|       |_____|
	//                    .           __|__
	//                   Go          | Gen |
	//                 Channel       |  G  |
	//                    .          |_____|
	//                    .             ^
	//                  __v__           .
	//                 |     |          . ARP, ICMP etc.
	//                 | Gen |         Go
	//                 |_____|       Channel
	//    _____         __|__         __.__         _____
	//   |     |       |     |    |o-|     |       |     |
	//   |  S0 |<-----<|  M  |<-----<|  H  |<-----<|  R1 |
	//   |_____|       |_____|       |_____|       |_____|

[leoluk]

That looks right - our pipeline is similar. The reason DPDK has lcores mapped to physical cores is to reduce latency by busy polling. Obviously not ideal for occasional ARP replies...

[leoluk]

Getting back to the original question... looking at the contributions graph, the project has either reached stability, or died:

Looks quite dead honestly. A shame Intel would drop it :(

[mikebromwich]

Yes what a shame. Especially given that we've just about finished our project to port our platform to us it! I'll have a look at Rust/Capsule - thanks for the suggestion.