Faster Python Serialization

[tokotchd]

Hello,

I see that there is great support for reading numpy arrays from flatbuffer vectors. I'm ecstatic that this is supported, but I could not find the reverse functionality.

In fact, It seems that the only way to write data to flatbuffer vectors in python is with builder.prependByte(). As you image, this is unusably slow for large data.

It seems like there could be a fairly easy and fast method of implementation that involves something akin to: builder.CreateByteVector(<numpy.ndarray>.tobytes()). Unfortunately, without hacking apart the library, there's assertions in place to prevent using this function from Python within the scope of another table or struct (nested assert), which is where it would need to be as I understand it.

@kbrose created the original functionality, is there a chance he'd be willing to work on this?

[aardappel]

@rw author of the Python port

[kbrose]

Hi @tokotchd! My work was solely about the reading of flatbuffers (I had no need to write in my use case), so unfortunately I will be unable to provide support on this work, but I wish you luck!

I will say that there may be a way to add the bytes in bulk with something like

builder.Bytes[builder.head : (builder.head + my_array.nbytes)] = my_array.tobytes()

making sure everything is in the correct byte order (little endian) and the builder has been correctly initialized (with StartVector I think?).

It would be great if this works out and that could be incorporated directly into the python API!

[tokotchd]

I haven't had much luck directly accessing builder.Bytes, when doing so I end up only writing len(my_array.tobytes()) bytes to the builder.

When I write with prependByte() and CreateByteVector() I end up writing about 30% more than expected bytes (8388608 instead of 6220804). I'm not sure where or why this overhead is coming from, even looking directly at source.

Can anyone elaborate?

[tokotchd]

The above overhead problem persists, but I have a working workaround thanks to @kbrose

bytesOfImage = testImage.tobytes()
Image.ImageStartDataVector(builder, len(bytesOfImage))
builder.Bytes[builder.head : (builder.head + len(bytesOfImage))] = bytesOfImage
data = builder.EndVector(len(bytesOfImage))

[kbrose]

Glad to hear it @tokotchd !

Do you think you could add an example of that to the documentation? @aardappel is that something you think would be useful?

[aardappel]

@kbrose sounds useful to me! Maybe as part of some special tips section in the python document?

[Amit072]

@tokotchd I think direct copy of data to Bytes buffer using below approach is not safe.

[1] builder.Bytes[builder.head : (builder.head + len(bytesOfImage))] = bytesOfImage

Prependbyte() internally calls Prep() which may take decision to enlarge the Bytes buffer with padding etc, and will update head if it decide to regrow the Byte buffer.

I think it is problematic to directly use approach [1] if we have followed up write operation to builder using Prependbyte.
And also, what if your image size is large than allotted Bytes array, you may overwrite the allotted byte buffer in builder without taking care of alignment, head etc.

I'm also looking for this optimization, PrependByte is too slow for large buffers.

[tokotchd]

@Amit072 Your points are valid, but I believe as long as you ensure that you prepare the data vector with the proper length using StartDataVector()... it should be okay.

That being said, you could always change your library to expose the Prep() function and call it yourself before writing directly to the bytes for additional safety. I'm unfortunately going to be not of much help besides the above solution as I have lost familiarity with the code and switched to using protobuffers for performance reasons.

[Amit072]

Thanks @tokotchd, I am able to make it working with below change. We just need to seek the header to correct location before writing to Bytes array.

 bytesOfImage = testImage.tobytes()
 Image.ImageStartDataVector(builder, len(bytesOfImage))
 #We need to seek the header to correct place before writing into Bytes array.
 builder.head = builder.head - len(bytesOfImage)

 builder.Bytes[builder.head : (builder.head + len(bytesOfImage))] = bytesOfImage
 data = builder.EndVector(len(bytesOfImage))

You can check my answer @stackoverflow
https://stackoverflow.com/questions/49239293/fast-python-serialize-to-flatbuffer?answertab=active#tab-top

[kbrose]

@Amit072 Do you think you could add an example of that to the documentation?

[gyenesvi]

This addition to the builder seems useful, however, one thing is not clear. If I have a data member in a table defined with forced alignment as:

table Buffer {
  data:[ubyte] (force_align: 16);
}

and I want to create the data array from numpy using data = builder.CreateNumpyVector(numpy_array), and then add the data with the generated method BufferAddData(builder, data) does it properly align the data? Isn't it a problem that we are bypassing the generated method BufferStartDataVector because of the use of builder.CreateNumpyVector ? I guess it depends on where the forced alignment is handled actually.

[felixfrank]

@gyenesvi, correct me if I'm wrong, but the way I read the flatbuffer docs and interpret the underlying idea of force alignment, is that it is an attribute of structs and cannot be used for arrays. As far as I know, if you add the attribute to a table field (like in your example) it is just ignored.

Also I'm curious, why would you actually want to add additional padding between individual array members of the same type?