Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Chunk::Max (chunk_max) and limit chunks to 2GB in auto modes #186

Open
aldanor opened this issue Nov 13, 2021 · 0 comments
Open

Add Chunk::Max (chunk_max) and limit chunks to 2GB in auto modes #186

aldanor opened this issue Nov 13, 2021 · 0 comments
Labels

Comments

@aldanor
Copy link
Owner

aldanor commented Nov 13, 2021

Since filter values are int32, the maximum safe chunk size is (1 < 31) - 1; e.g., blosc would fail otherwise. Should we limit chunk sizes by 2GB? (at least when using min-kb mode, or when determining chunk size automatically)

It might also be nice to add Chunk::Max which would pick the largest possible chunk size (i.e., shape), but limited by 2GB.

@aldanor aldanor changed the title Add Chunk::Max (chunk_max) and limits chunks to 2GB in auto modes Add Chunk::Max (chunk_max) and limit chunks to 2GB in auto modes Nov 19, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant