Initial JSON UDP Plugin Implementation

[hawkinsw]

This is a simple plugin that will format captured
records as JSON objects and then deliver them
to a listening UDP server.

[hawkinsw]

Hello! pmacct is such a great set of programs and a great tool. We are using it in our research project and had the requirement that we format netflow records as json-objects as part of a data processing pipeline. This little plugin does exactly what we need -- it formats each record as a json object and then sends it to a server over UDP.

I would love for this to be included if you think that it's useful. I know that there will be things that you want to fix -- please let me know how I can modify the patch so that it meets your high standards.

Thanks again for the great work maintaining and developing such a great suite of tools!

Will

[paololucente]

Dear Will / @whh8b ,

Thank you very much for your contribution - it is very much appreciated. The code looks sane and consistent. What i did particularly like is that the plugin does not perform any caching so it is a nice shim layer for at least two use-cases: 1) exporting data to a 3rd party tool without passing through any complex broker-based systems (ie. Kafka, RabbitMQ) and 2) as an enabler/code hook to write an actual pmacct plugin in a different language than C/C++.

This all said, and as we did chatted about by email, i would see a huge benefit in introducing ZeroMQ as output - in addition or replacement to the current UDP-based one: on one hand, i see benefits in having the simplest output possible (that is, UDP), on the other hand i see the cons of being unreliable and potential issues with maximum message size (and hence the need for extra complexity, that is, implementing message segmentation). Another potential area to look at could be features to (optionally) optimise output: binary encodings, ie. Apache Avro comes to mind, batching records, ie. one message per record is not very efficient, or compressing JSON.

Given the two distinct use-cases and the very much wanted introduction of ZeroMQ, maybe it could also make sense to 'brand' the plugin with a different name; cosmetic, we can think at it together.

To wrap-up, i support the initiative and would very much like if ZeroMQ output could be also made part of the initial plugin implementation. From that point onwards we can define a bit of a roadmap of which features to add.

Regards,
Paolo

[hawkinsw]

Just FYI: As we emailed about last night, here is an updated pull request for the plugin that includes zeromq support! I hope that this matches your expectations!

As we also talked about in the email, I am waiting on the rebranding to hear your creative thoughts on the new name :-)

I look forward to working together to get this into shape so that it matches your high-quality standards!