read and write sparse matrix with intersection weights using xarray

[cchwala]

The calculation of the sparse intersection weights via calc_intersect_weights is fairly slow (approx. 4 minutes for 4000 CMLs and the 900x900 RADOLAN grid) and the resulting weight matrix would be large in dense format. Hence, it is often required to write the sparse intersection weights to disk and read them back in later to reuse them. Writing sparse data to a file e.g. in COO format is possible, but we want to keep the linkage to the indices that we have in the xarray.DataArray of the sparse weights matrix.

What we need is a function to read and write a xarray.DataArray, containing a sparse matrix, to NetCDF or zarr.

[cchwala]

There is a solution that could actually cover our use case in pangeo-data/xESMF#222 (comment)

[cchwala]

At the end of this notebook Sparse arrays and the CESM land model component I learned that there is also a CF convention for storing sparse data via Compression by Gathering. Not sure if this helps us. But there is some implementation, at least for MultiIndex for this in cf-xarray according to their docs using cfxr.encode_multi_index_as_compress and cfxr.decode_compress_to_multi_index.

[cchwala]

A simple solution is to do something like

sparse.save_npz('sparse_intersect_weights.npz', sparse_intersect_weights)

and store the coordinates of the xarray.DataArray separately

Then one can load like this

 def load_sparse_intersect_weights():
    ds_intersect_weights = xr.open_dataset('intersect_weights_coords.nc')
    da_intersect_weights = xr.DataArray(
        data = sparse.load_npz('sparse_intersect_weights.npz'),
        coords = ds_intersect_weights.coords,
    )
        
    return da_intersect_weights

da_intersect_weights = load_sparse_intersect_weights()
da_intersect_weights