Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add option to use different s3 settings for inputs and output #323

Open
ungarj opened this issue Mar 1, 2021 · 0 comments
Open

add option to use different s3 settings for inputs and output #323

ungarj opened this issue Mar 1, 2021 · 0 comments

Comments

@ungarj
Copy link
Owner

ungarj commented Mar 1, 2021

e.g. to enable reading and writing using different access keys or even s3 backends

...
input:
  s2:
    format: S2AWS_COG
    start_time: 2018-04-02
    end_time: 2018-04-03
    read_options:  # for input drivers using rasterio and/or fiona, this mapping should be passed on in rasterio.Env or fiona.Env
        AWS_ACCESS_KEY_ID: ENV.SRC_AWS_ACCESS_KEY_ID  # read SRC_AWS_ACCESS_KEY_ID from environment variable
        AWS_DEFAULT_REGION="eu-central-1"  # use this string
        AWS_SECRET_ACCESS_KEY=ENV.SRC_AWS_SECRET_ACCESS_KEY  # read SRC_AWS_SECRET_ACCESS_KEY from environment variable
...
output:
  format: GTiff
  bands: 3
  path: s3://some-bucket/
  dtype: uint8
  write_options:  # for output drivers using rasterio and/or fiona, this mapping should be passed on in rasterio.Env or fiona.Env
      AWS_ACCESS_KEY_ID: ENV.DST_AWS_ACCESS_KEY_ID  # read DST_AWS_ACCESS_KEY_ID from environment variable
      AWS_S3_ENDPOINT_URL="https://obs.some-provider.com/"  # use this string
      AWS_SECRET_ACCESS_KEY=ENV.DST_SECRET_ACCESS_KEY  # read DST_SECRET_ACCESS_KEY from environment variable
...

This should work for drivers using rasterio and/or fiona but we have to make sure, these mappings are passed on to every rasterio/fiona call. Likewise drivers using different packages like xarray should be provided also with this mapping if possible. Hence the generic names read_options and write_options.

Note that it should be possible by either providing these settings in the configuration as raw string or as a reference to an environment variable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant