pyramid_reroject OOM'ing

[norlandrhagen]

When creating pyramids for the AGDC dataset, pyramid_reproject was leading to OOM errors. This was happening using both Distributed and vanilla dask.

Example for later debugging:

from ndpyramid import pyramid_reproject
import rioxarray
import xarray as xr 
import distributed


from dask.distributed import Client
client = Client(n_workers=4)
client 

store = 'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/pangeo-forge/AGDC-feedstock/AGCD.zarr'
ds = xr.open_dataset(store, engine='zarr', chunks={})

ds = ds.rio.write_crs('EPSG:4326')
pyramid = pyramid_reproject(ds,levels=4)