Use Intake

[martindurant]

A light suggestion that people here may want to consider expressing this collection of data-sets using Intake, which is a cataloging standard allowing you to describe the metadata and way-to-load for data in a single browsable, searchable spec, e.g., a YAML file. This is, for instance, what the pangeo collaboration does. Most of their data is in zarr or other xarray-compatible formats, so you would also need intake-astro to enable loading FITS data from remote files, with local caching if desired, and lazy-loading with Dask.

[pllim]

Thank you for the suggestion! Unfortunately, the current maintainers are stretched pretty thin to investigate this option. If you are interested to implement a proof-of-concept for this new solution, we would be happy to review your PR and continue the discussion there.

[pllim]

Looks like scipy uses pooch now but even that seems a little heavyweight for our purpose here. 🤷