Stackstac on AWS EC2

[vitusbenson]

Hi @gjoseph92 ,

in the Readme there is this line:

stackstac/README.md

Line 93 in 0dff7cf

- Easier access to `s3://`-style URIs (right now, you'll need to pass in `gdal_env=stackstac.DEFAULT_GDAL_ENV.updated(always=dict(session=rio.session.AWSSession(...)))`)

I am trying to figure out how to best use stackstac in an AWS instance, for example for reading Sentinel 2 COGs. I realize the pystac_client gives only http: urls and not the s3:. Was wondering if gdal_env solves that or if I would have to manually fiddle with the stac items.

I am aware of the great work you do with coiled, however due to billing issues it is not an option for this project - that is why I am trying to do this directly in AWS. The idea is run an instance that is in the same region as the data and thus get high read speed. Unfortunately I currently just see an increase of 2x compared to running locally, which confuses me.

Do you have any tips?

[gjoseph92]

Hey @vitusbenson, I'm not quite sure what you're asking?

I am trying to figure out how to best use stackstac in an AWS instance, for example for reading Sentinel 2 COGs

Just an idle question, I'm also curious why you wouldn't use Planetary Computer for this. They make it pretty easy and give you a bit of free compute too.

I realize the pystac_client gives only http: urls and not the s3:. Was wondering if gdal_env solves that or if I would have to manually fiddle with the stac items.

You don't need to use s3:// URLs. It's probably easier if you just don't. AFAIU, S3-style URLs are just syntactic sugar that AWS clients convert into HTTP URLs internally and don't have any performance benefit—S3 only has an HTTP API.

The idea is run an instance that is in the same region as the data and thus get high read speed

This seems reasonable and like normal usage.

Unfortunately I currently just see an increase of 2x compared to running locally, which confuses me

I'd need more information to diagnose this. You might have a different number of cores/workers locally vs on the VM, different amount of memory, etc.

[vitusbenson]

Hi @TomAugspurger , sorry I don't have time right now to produce a minimal working example for you, will get back to you later. However I am catching the rate limits currently like this (which of course slows the code significantly):

search = catalog.search(
                        bbox = bbox,
                        collections=["sentinel-2-l2a"],
                        datetime=time_interval
                    )

for attempt in range(10):
    try:
        items_s2 = pc.sign(search)
    except pystac_client.exceptions.APIError:
        print(f"Sen2: Planetary computer time out, attempt {attempt}, retrying in 60 seconds...")
        time.sleep(random.uniform(30,90))
    else:
        break
else:
    print("Loading Sen2 failed after 10 attempts...")

[TomAugspurger]

Thanks. Are you doing that in parallel from many machines? The planetary_computer package will cache tokens per storage container, so I'm a bit surprised that it's affecting you.

Actually... why would pc.sign be throwing a pystac_client.exceptions.APIError? That makes me think the issue is actually in something like search.get_all_items(), which might be being called by the planetary_computer.sign.

[vitusbenson]

Yes, it was pystac_client.exceptions.APIError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')). It came when running ~40 processes in parallel on one machine that were all trying to access the STAC from planetary_computer.

[TomAugspurger]

Ah, that's a service issue then, not rate limiting.

You might want to subscribe to microsoft/PlanetaryComputer#68. We're still debugging it, but don't have a satisfying answer to what's causing it yet.

[vitusbenson]

i see, thanks!

[0-ren]

hey @vitusbenson I'd love to learn more about billing issue means re: coiled.

All users get to use coiled for free for their first 10K cpu hours. Our pricing is being reworked at the moment, so I understand why this might be confusing.

Are you free sometime this week to talk it over? Here's a link to my calendar where you can find a time that works best for you. Here to help! Thanks!

[vitusbenson]

Hi @0-ren oh cool, I did not know this, thanks. Now I started trying out coiled, but am running into many unkown errors - which is another issue:)
Re billing: I am with German public research institute. For this project we basically got some AWS credits to use. In general, it is just very hard for us to spend significant amounts on firms without having a public tender...
Thanks for the offer to chat, I am not sure if it would be beneficial, what do you think?

[gjoseph92]

When you use coiled you both pay AWS for the instances, and Coiled for the management of them. You'd still be using your AWS credits for the instances. And Coiled's free tier might mean that the management is free as well, depending on your usage.

[0-ren]

@vitusbenson What errors are you running into? Feel free to post them here - https://github.com/coiled/feedback/issues
And hopefully you should have enough credits with Coiled's free credits to play around with all the AWS ones you got for this project.

[BrianAvant]

@gjoseph92 I would also like to call stackstac with a modified gdal_env parameter. I trying to access images from an S3 bucket and I am passing the param as:
gdal_env=stackstac.DEFAULT_GDAL_ENV.updated(always=dict(session=rio.session.AWSSession(boto3.Session(),requester_pays=True)))

but I get a TypeError: "Could not serialize object of type LayeredEnv". If I leave the default gdal_env I get RasterIOError('Access Denied').

I was looking at rio_reader.py and tried to replicate the _open function

gdal_env=stackstac.DEFAULT_GDAL_ENV.updated(always=dict(session=rio.session.AWSSession(boto3.Session(),requester_pays=True)))
with gdal_env.open:
       ds = rio.DatasetReader(image_S3_uri, sharing=False)
       ds_data = ds.read()

I am able to access the data this way.

[gjoseph92]

Great! That seems like a better way to do it.

[BrianAvant]

I have a new issue where I am running this same call but using a Dask gateway cluster. For some reason I am getting the same RasterIOError('Access Denied') error but only when running on a gateway cluster. Running locally or with a LocalCluster works fine.

I have been using the method we discussed earlier where I just update the DEFAULT_GDAL_ENV and pass it to stackstac.stack as the gdal_env arg.

gdal_env_vars = stackstac.DEFAULT_GDAL_ENV.updated(
      always=dict(AWS_REQUEST_PAYER="requester")
)

I also tried updating the default dummy session to an AWSSession

gdal_env_vars.open.session = AWSSession(boto3.Session(),requester_pays=True)

Even applied this to always, open, read, and open_vrt with no luck resolving the error. Any idea what is going on here? I am delaying dask compute until writing to a zarr file xr.to_zarr()

Here is the complete traceback

Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/stackstac/rio_reader.py", line 326, in _open
ds = SelfCleaningDatasetReader(
File "rasterio/_base.pyx", line 304, in rasterio._base.DatasetBase.__init__
rasterio.errors.RasterioIOError: Access Denied

The above exception was the direct cause of the following exception:
File "/usr/local/lib/python3.8/site-packages/xarray/core/dataset.py", line 2060, in to_zarr
  return to_zarr(  # type: ignore
File "/usr/local/lib/python3.8/site-packages/xarray/backends/api.py", line 1638, in to_zarr
  writes = writer.sync(compute=compute)
File "/usr/local/lib/python3.8/site-packages/xarray/backends/common.py", line 168, in sync
  delayed_store = da.store(
File "/usr/local/lib/python3.8/site-packages/dask/array/core.py", line 1230, in store
  compute_as_if_collection(Array, store_dsk, map_keys, **kwargs)
File "/usr/local/lib/python3.8/site-packages/dask/base.py", line 342, in compute_as_if_collection
  return schedule(dsk2, keys, **kwargs)
File "/usr/local/lib/python3.8/site-packages/distributed/client.py", line 3036, in get
  results = self.gather(packed, asynchronous=asynchronous, direct=direct)
File "/usr/local/lib/python3.8/site-packages/distributed/client.py", line 2210, in gather
  return self.sync(
File "/usr/local/lib/python3.8/site-packages/distributed/utils.py", line 338, in sync
  return sync(
File "/usr/local/lib/python3.8/site-packages/distributed/utils.py", line 405, in sync
  raise exc.with_traceback(tb)
File "/usr/local/lib/python3.8/site-packages/distributed/utils.py", line 378, in f
  result = yield future
File "/usr/local/lib/python3.8/site-packages/tornado/gen.py", line 762, in run
   value = future.result()
File "/usr/local/lib/python3.8/site-packages/distributed/client.py", line 2073, in _gather
     raise exception.with_traceback(traceback)
File "/usr/local/lib/python3.8/site-packages/dask/optimization.py", line 990, in __call__
      return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
File "/usr/local/lib/python3.8/site-packages/dask/core.py", line 149, in get
    result = _execute_task(task, cache)
File "/usr/local/lib/python3.8/site-packages/dask/core.py", line 119, in _execute_task
      return func(*(_execute_task(a, cache) for a in args))
File "/usr/local/lib/python3.8/site-packages/stackstac/to_dask.py", line 185, in fetch_raster_window
      data = reader.read(current_window)
File "/usr/local/lib/python3.8/site-packages/stackstac/rio_reader.py", line 385, in read
      reader = self.dataset
File "/usr/local/lib/python3.8/site-packages/stackstac/rio_reader.py", line 381, in dataset
      self._dataset = self._open()
File "/usr/local/lib/python3.8/site-packages/stackstac/rio_reader.py", line 337, in _open
      raise RuntimeError(msg) from e
RuntimeError: Error opening 's3://<pathtoimage>.tif': RasterioIOError('Access Denied')
      py.warnings: /usr/local/lib/python3.8/site-packages/dask_gateway/client.py:1014: RuntimeWarning: coroutine 'rpc.close_rpc' was never awaited
      self.scheduler_comm.close_rpc()

[gjoseph92]

I am getting the same RasterIOError('Access Denied') error but only when running on a gateway cluster

I would assume that there are no AWS credentials available on your gateway cluster. If the dask workers don't know what AWS account to use, who's the requester that's gonna pay?

[gjoseph92]

It certainly seems like forwarding credentials to the cluster is a pain point with stackstac. However, I'm not sure I'd like to make it part of stackstac's responsibility. It's a pretty general problem that comes up when trying to access any type of data from a cloud provider.

If you use Coiled to deploy your cluster, your AWS credentials will automatically be securely forwarded to the cluster: https://docs.coiled.io/user_guide/security.html#aws-credentials. Then the code you have that already works locally should just work on the cluster.

I'm not aware of other off-the-shelf solutions for credential forwarding to Dask clusters. You'd probably need to generate an STS token yourself in AWS, then set it as another environment variable that GDAL can pick up in order to authenticate. Note that this will be passed around the cluster and in your code in the clear, so be sure you trust your infrastructure, and don't accidentally commit it to version control.

[BrianAvant]

You nailed it. Just needed to pass AWS credentials to os.environ and it worked. Thanks again!