Arrow c data interface

[Alex-PLACET]

Fix #69

[paleolimbot]

Very cool! Just some notes from writing a few things along these lines in nanoarrow 🙂

[paleolimbot]

I would highly recommend avoiding std::unique_ptr<ArrowSchema/ArrowArray>, unless used with a deleter that ensures that any unreleased array that it is pointing to is released when the unique pointer is deleted. In cudf they use a unique pointer with a deleter:

https://github.com/rapidsai/cudf/blob/e58838b6cc820fc89f1f67eb9117a3ee6ddeaa47/cpp/src/interop/to_arrow_device.cu#L601-L604

...in nanoarrow we have a UniqueArray/Schema class that wraps a stack-allocated ArrowArray/Schema:

https://github.com/apache/arrow-nanoarrow/blob/a0632177c27cb52880c62533fcd441613f41815b/src/nanoarrow/nanoarrow.hpp#L265-L272

..but the idea is the same: always make a home for the struct such that it is guaranteed to be cleaned up, and "move" it between safe homes (e.g., using something like https://github.com/apache/arrow-nanoarrow/blob/a0632177c27cb52880c62533fcd441613f41815b/src/nanoarrow/nanoarrow_types.h#L334-L340 ).

(Apologies if there's some C++ magic going on here that already ensures this)

[paleolimbot]

All of the implementations of an ArrowArray/ArrowSchema I've read choose to own their own memory for the ArrowArray/ArrowSchema structs. In this case, it might mean an std::move() of the std::vector<std::unique_ptr<ArrowArray>> into the private data and rely on the C++ deleter to call the release callbacks (see my comment below about always making sure that std::unique_ptr<ArrowThing> is guaranteed to call the release callback if present).

Alternatively, one could use new ArrowArray[] and something like https://github.com/apache/arrow-nanoarrow/blob/a0632177c27cb52880c62533fcd441613f41815b/src/nanoarrow/nanoarrow_types.h#L334-L340 to "move" the struct from foreign memory.