How important is the use of virtual memory for an image storage format? #4
Replies: 3 comments 1 reply
-
@ neurolabusc at November 16, 2021, 3:41pm @ matthew-brett - this is somewhat related to my recent work supporting the probabilistic Julich-Brain-Atlas in MRIcroGL and NiiVue. The raw data is in a simple NIfTI format, but requires 9.4Gb when decompressed and access is not cache friendly. A combination of a NIfTI offset table and a lookup table reduces this by a factor of x422. Since probabilistic atlases are rare, perhaps no one else is interested in this topic. However, if anyone has suggestions for the scheme (or are interested in adopting it), I would be interested to hear from them. The labels could be HAWG compliant, just not using the classic NIfTI storage. |
Beta Was this translation helpful? Give feedback.
-
@ oesteban at November 16, 2021, 4:17pm Hi @ matthew-brett - this looks like a great idea to me. I would try to attempt some sort of unification with surface formats. In the end, both are composed of the same elements: data, location of those datapoints, and the relationship between them (faces for surfaces and a grid definition for images). It would be valuable not just having fast and lazy loading across all dimensions, not just the last. This would be really useful when extracting ROIs or when dividing a volume into sub-images for parallel processing. FWIW, in the context of fMRIPrep I've been thinking a lot about having some internal HDF5 specification and only use NIfTI/GIFTI/CIFTI when writing the outputs. In dMRIPrep things might be even closer to that model. I would be very happy if I could use something tailored to the problem, rather than HDF5. @ neurolabusc - I would say that the issue is not that people are not interested in probabilistic atlases - I'd say it goes the other way around (hard to share, integrate and access → let's use something simpler) |
Beta Was this translation helpful? Give feedback.
-
@ pesto at November 16, 2021, 4:35pm Great idea. Do you think that it would be feasible to extend the discussion to be major modalities? For example:
It sounds like @ oesteban already suggested extending to Surface data. @ arokem @ rheault and @ garyfallidis (not here yet) and I have been discussing a format for Tractography in DIPY Franco |
Beta Was this translation helpful? Give feedback.
-
November 12, 2021, 1:07pm
Neuroimages are large, and may not fit into memory. We would therefore like a format that allows array-like access to a large data file, without loading the whole file into memory. I'm calling this virtual memory.
How important is the ability to use virtual memory, an image format?
I think we need to distinguish between:
For sharing formats - we probably don't need virtual memory access, but I think virtual memory access is probably essential for an analysis format. What do y'all think?
Beta Was this translation helpful? Give feedback.
All reactions