Question: Variable length element sizes?

[vectronic]

Use cases of receiving multiplexed video packets or compressed video frames relies on supporting variable length data elements. I assume the only way to support this in Raft would be to have a fixed size wrapper element in the FIFOs and again point to a global memory pool which supports variable buffer sizes or to pre-allocate the elements with the maximum possible variable size; and subsequentally waste memory?

An example is Key Length Value (KLV) packets. In this case a fixed number of Key bytes indicates the type of element and the Length indicates the length of the Value. In this case ideally the element type being passed to a kernel is a KLV, but the size of this element can be variable.

[jonathan-beard]

This one requires a bit more thought. I think the pool answer above gives the answer when using a wrapper. Given that C++ is strongly typed, this goes against that grain a bit, essentially allowing a type to have a variable storage size. We don't necessarily have to get this into the standard so we can perhaps fudge a bit on the strongly typed nature as long as we can guarantee that the stream/FIFO is all of a base type even if the encoded data is a bit longer.

With KLV packets, there's still an implicit size for the most part, that ends up being the unit over which you're traversing. If you look at bus packets for instance, you typically have to send in units of bus width (think 64-bits minimum). If we think of allocations, the min unit I can often get is a 4K page.

Here's a solution that could work depending on the use case and patterns (for that, could you provide a short example?):

1. Wrapper port type that enables solidifies the "signal" provided by RaftLib ports as a means to transmit the key/length to consume from a separate channel. We wouldn't even have to provide a wrapper necessarily, however, it'd provide semantics to show the usage vs. just a one off usage that programmers would have to figure out for every usage.
2. Use memory pools with the variable size packets (e.g., struct with key/length + pointer to buffer).
3. Any other suggestions? I like number one the best as it'd be the easiest to work with.

[vectronic]

Can you point me anywhere at an example of a wrapper port type you mention so that I can think how it might be used? Or is that a new concept you were suggesting?

A simple usage example of KLV might be a tool that scans the file and outputs the important characteristics of a media file e.g. number of audio and video tracks, duration, audio channels, sample rate etc. All of this information is encoded in the file in specific KLV entries. The tool would need to scan through the sequential KLV items (including hierarchies of KLV containers which have child KLV entries) and look out for the specific KLV items which have the known K values which indicate the characteristics wanted appear in the corresponding V.

I guess you could think of it like a KLV stream where you define a K filter and then a V handler.