Techniques for Efficiently Handling FastMRI Dataset

[stperrakis]

Hello,

in case of space limitations, Is there any efficient way to work with data without having to unzip the batch files?
Is there any way to avoid buying extra space?

Thanks in advance.

[mmuckley]

I think if you were to recompress as .zip files you could read individual files directly from the zip file. https://stackoverflow.com/questions/19371860/python-open-file-in-zip-without-temporarily-extracting-it

There might be a way to do this with .tar.xz as well.

Either one of these will slow down data loading speed. The batch files themselves were meant to be the primary solution to this problem. Previously, you could only work with the entire dataset.

[stperrakis]

While it is recommended to work with each batch separately, it should be noted that each uncompressed batch is approximately 1+ terabyte in size. In my situation, this can be difficult even when processing individual batches. As accessing specific .h5 file within a batch, it may take several minutes due to high dense compression.

[mmuckley]

@stperrakis it depends on the compression you use. .xz compression is very dense, as you said, but you could recompress the dataset with a less dense level of compression. Note there are other small datasets out there with less data than one of the fastMRI batches that you could also use.

Still, I could see some use for selecting an even smaller tiny fastMRI dataset. I've opened Issue #305 to collect interest. Please emoji react there if you would like this implemented.