Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Tensorstore support #242

Closed
wants to merge 3 commits into from
Closed

Add Tensorstore support #242

wants to merge 3 commits into from

Conversation

jacobbieker
Copy link
Member

Pull Request

Description

This adds optionally opening zarrs with Tensorstore, which can be up to 10-40x faster than opening Zarrs encoded with ocf_blosc2. This comes with a size penality, in that Zarrs are then around 28% larger, but since speed of reading Zarrs is a large bottleneck for us, probably worth the tradeoff. In either case, the tensorstore backend can be ignored, as Tensorstore only supports limited compression algorithms.

Fixes #224
Fixes #198

How Has This Been Tested?

Unit tests

  • Yes

If your changes affect data processing, have you plotted any changes? i.e. have you done a quick sanity check?

  • Yes

Checklist:

  • My code follows OCF's coding style guidelines
  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • I have checked my code and corrected any misspellings

@jacobbieker jacobbieker self-assigned this Nov 17, 2023
@jacobbieker
Copy link
Member Author

Closing this as @dfulu did some benchmarking, and it seems like this would have a minimal effect on loading speeds, while requiring that we re-compress our data again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Test out Xarray Tensorstore Backend Add support for TensorStore for Zarr opening
1 participant