Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3 Grib support #1177

Open
tdrwenski opened this issue May 9, 2023 · 1 comment
Open

S3 Grib support #1177

tdrwenski opened this issue May 9, 2023 · 1 comment
Labels
enhancement New feature or request

Comments

@tdrwenski
Copy link
Member

tdrwenski commented May 9, 2023

Currently, using GRIB files in an ObjectStore don't work correctly due to the index files (.gbx9) being assumed to be local files. No index file is created if the expected one is not found, since it tries to create a directory using the s3 path which fails.

Expected behavior:

  • use or update ObjectStore index file if available
  • use local cache (if that setting is turned on) for storing the index file
  • create or update the index file in the ObjectStore next to the data files if the cache is not turned on

The first and last bullet require creating or updating objects in the ObjectStore, which can be handled through MFile. We could also choose to start with only using a local cache for index files.

@WeatherGod
Copy link
Contributor

Another thing to consider is that (in my opinion) an S3 source of grib files will be from places like ECMWF or one of the Open Data buckets, which is read-only for end-users.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants