How do I store cell boundaries with varying number of points?

[Rushil-Chakra]

Hello,

I have a zarr dataset of 100k cell boundaries that range in the number of points per polygon, going up to ~400 for a single cell. How does Vitessce expect this data to be stored? I tried creating an ndarray with shape (n_cells, max_points, 2) in the obsm and padding the smaller shapes with None, but this raised an error: Buffer has NaN of dtype |O.

Thanks!

[ilan-gold]

If I'm not mistaken this is basically not allowed. I don't think AnnData supports ragged arrays (which are what would be needed for variable length per-obs vectors as you are suggesting) and if they do, we do not at the moment. So if/when they do, we will make it a more of a priority. It seems like it's ongoing work for them (PR 647 in their repo). FWIW a 400 point polygon sounds very high - OME metadata bitmasks are very well supported in Vitessce.

[keller-mark]

That is correct, currently all of our data loaders for arrays assume a uniform shape. However, you could implement a plugin file type that handles loading a ragged array and assumes some particular padding value. Currently we do not have documentation about the data representation that such a data loader would need to return, but in #1261 we are changing the internal representation of segmentation polygon data slightly so you may want to wait for us to merge that pull request before implementing a plugin. With that pull request, we will also be adding documentation about the internal representation for each data type.

That being said, if you want to implement a plugin data loader based on the current data representation, it is a JSON object where keys are cell IDs and values are JSON objects containing the key poly to store the polygon vertices. (for 100k cells with 400-vertex polygons this JSON object might get pretty large, which is another motivation behind changing our internal data representation to use more efficient data structures)

[Rushil-Chakra]

Thanks for the resources! I'll definitely look into them. I originally had a JSON object but was only able to load half the dataset with it. I agree that 400 points is a lot and am looking into reducing shape resolution to continue trying with the JSON object.