Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow for single partial chunk #73

Open
rsignell-usgs opened this issue Dec 16, 2020 · 4 comments
Open

Allow for single partial chunk #73

rsignell-usgs opened this issue Dec 16, 2020 · 4 comments

Comments

@rsignell-usgs
Copy link
Member

When I create a large rechunked dataset, I use rechunker on each time chunk, then append. The last time chunk currently fails because rechunker thinks the chunk plan is invalid (because only a single partial chunk is being written).

Here's a full example: https://nbviewer.jupyter.org/gist/rsignell-usgs/c0b87ed1fa5fc694e665fb789e8381bb

@rsignell-usgs rsignell-usgs changed the title Allow for partial chunk Allow for single partial chunk Dec 16, 2020
@rabernat
Copy link
Member

This is what @davidbrochart was working on in #28. That stalled because we did some internal refactoring which broke his approach. But maybe we can convince him to pick it up again? 😊

@davidbrochart
Copy link
Collaborator

I'd be happy to work on this, but didn't have time so far 😞

@rabernat
Copy link
Member

I wish we could argue that this is within scope for the Quantstack Zarr EOSS work! 😆

@davidbrochart
Copy link
Collaborator

Me too! But not related to C++ or Zarr v3 unfortunately.
If QuantStack could be involved in another Zarr project, I'd love to work on these things.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants