[Draft] Non-kerchunk backend for HDF5/netcdf4 files. #87
Conversation
There was a problem hiding this comment.
This is looking great so far @sharkinsspatial !
kerchunk backend's specialized encoding translation logic
This part I would really like to either factor out, or at a least really understand what it's doing. See #68
virtualizarr/readers/hdf.py
Outdated
| @@ -0,0 +1,206 @@ | |||
| from typing import List, Mapping, Optional | |||
|
|
|||
| import fsspec | |||
There was a problem hiding this comment.
Does one need fsspec if reading a local file? Is there any other way to read from S3 without fsspec at all?
There was a problem hiding this comment.
Not with a filesystem-like API. You would have to use boto3 or aiobotocore directly.
This is one of the great virtues of fsspec and is not to be under-valued.
virtualizarr/readers/hdf.py
Outdated
| def virtual_vars_from_hdf( | ||
| path: str, | ||
| drop_variables: Optional[List[str]] = None, | ||
| ) -> Mapping[str, xr.Variable]: |
There was a problem hiding this comment.
I like this an a way to interface with the code in open_virtual_dataset
|
This looks cool @sharkinsspatial! My opinion is that it doesn't make sense to just forklift the kerchunk code into virtualizarr. What I would love to see is an extremely tight, strictly typed, unit-tested total refactor of the parsing logic. I think you're headed down the right path, but I encourage you to push as far as you can in that direction. |
TomNicholas
added
enhancement
for more information, see https://pre-commit.ci
|
@rabernat Fully agree with your take above 👆 👍 . I'm trying to work through this incrementally whenever I can find some spare time. In the spirit of thorough test coverage 🎊 looking through your issue pydata/xarray#7388 and the corresponding PR I'm not sure what the proper incantation of variable encoding configuration is to use |
for more information, see https://pre-commit.ci
This is a rudimentary initial implementation for #78. The core code is ported directly from kerchunk's hdf backend. I have not ported the bulk of the kerchunk backend's specialized encoding translation logic but I'll try to do so incrementally so that we can build complete test coverage for the many edge cases it currently covers.