access Sentinel-1 IW SLC tiff measurement via S3 on creodias entrypoint

[agrouaze]

Expected behavior and actual behavior.

I expected to read a band from a file and an exception occurred:

raceback (most recent call last):
  File "rasterio/_base.pyx", line 308, in rasterio._base.DatasetBase.__init__
  File "rasterio/_base.pyx", line 219, in rasterio._base.open_dataset
  File "rasterio/_err.pyx", line 221, in rasterio._err.exc_wrap_pointer
rasterio._err.CPLE_AppDefinedError: JSON parsing error: continue (at offset 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home1/datahome/agrouaze/sources/test_rasterio/test_s1_eodata_s3_access.py", line 26, in <module>
    with rasterio.open(s3_key) as src:
  File "/opt/conda-envs/dev/lib/python3.10/site-packages/rasterio/env.py", line 451, in wrapper
    return f(*args, **kwds)
  File "/opt/conda-envs/dev/lib/python3.10/site-packages/rasterio/__init__.py", line 304, in open
    dataset = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)
  File "rasterio/_base.pyx", line 310, in rasterio._base.DatasetBase.__init__
rasterio.errors.RasterioIOError: JSON parsing error: continue (at offset 0)

Steps to reproduce the problem.

import s3fs
import pdb
import rasterio

access_key = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
secret_key = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
entrypoint_url = 'eodata.dataspace.copernicus.eu'

# Open the raster file from S3 using Rasterio
s3_key = 's3:///eodata/Sentinel-1/SAR/IW_SLC__1S/2023/02/28/S1A_IW_SLC__1SDV_20230228T041847_20230228T041916_047431_05B1B4_B985.SAFE/measurement/s1a-iw1-slc-vv-20230228t041847-20230228t041915-047431-05b1b4-004.tiff'
session = s3fs.S3FileSystem(key=access_key, secret=secret_key)
with rasterio.Env(aws_session=session):
    with rasterio.open(s3_key) as src:
        print(src.profile)
        data = src.read(1)  # Read the first band
        pdb.set_trace()

Environment Information

• rasterio version 1.3.6
• GDAL version 3.6.2
• Python version 3.10.8 | packaged by conda-forge | (main, Nov 22 2022, 08:26:04) [GCC 10.4.0]
• Operation System Information Linux-6.5.0-25-generic-x86_64-with-glibc2.35

Installation Method

micromamba install -c conda-forge rasterio

[sgillies]

session = s3fs.S3FileSystem(key=access_key, secret=secret_key)
with rasterio.Env(aws_session=session):

@agrouaze this is invalid usage. The value of aws_session must be a boto3 Session instance.

[agrouaze]

Hi @sgillies ,
I corrected the snippet like this:

session = boto3.Session(
    aws_access_key_id=access_key,
    aws_secret_access_key=secret_key
)
# Open the raster file from S3 using Rasterio
s3_key = 's3:///eodata/Sentinel-1/SAR/IW_SLC__1S/2023/02/28/S1A_IW_SLC__1SDV_20230228T041847_20230228T041916_047431_05B1B4_B985.SAFE/measurement/s1a-iw1-slc-vv-20230228t041847-20230228t041915-047431-05b1b4-004.tiff'
#session = s3fs.S3FileSystem(key=access_key, secret=secret_key)
with rasterio.Env(aws_session=session):
    with rasterio.open(s3_key) as src:
        print(src.profile)
        data = src.read(1)  # Read the first band
        pdb.set_trace()

but it failed with this message:

Traceback (most recent call last):
  File "rasterio/_base.pyx", line 310, in rasterio._base.DatasetBase.__init__
  File "rasterio/_base.pyx", line 221, in rasterio._base.open_dataset
  File "rasterio/_err.pyx", line 221, in rasterio._err.exc_wrap_pointer
rasterio._err.CPLE_AppDefinedError: JSON parsing error: continue (at offset 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/antoine/Documents/sources/test_rasterio/read_tiff_with_rasterio_through_s3.py", line 20, in <module>
    with rasterio.open(s3_key) as src:
  File "/home/antoine/micromamba/envs/micromamba1/lib/python3.10/site-packages/rasterio/env.py", line 451, in wrapper
    return f(*args, **kwds)
  File "/home/antoine/micromamba/envs/micromamba1/lib/python3.10/site-packages/rasterio/__init__.py", line 304, in open
    dataset = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)
  File "rasterio/_base.pyx", line 312, in rasterio._base.DatasetBase.__init__
rasterio.errors.RasterioIOError: JSON parsing error: continue (at offset 0)

[sgillies]

@agrouaze I'm sorry you're not succeeding. Two things

1. The s3:///eodata/... looks suspicious to me. Generally we make S3 identifiers like s3://BUCKETNAME/KEY. Two slashes only.
2. If I am not mistaken the error comes from https://github.com/OSGeo/gdal/blob/6d3b13a95c8c0ad042b4031483da67c6a307cc02/port/cpl_json.cpp#L270. The message isn't very helpful but hints at GDAL trying to parse something as JSON which is not JSON.

If you haven't already found a solution, I recommend running your program with CPL_CURL_VERBOSE=ON in the environment. This will cause GDAL to emit a lot of extra logging and you may see in it signs of a failed AWS request or unexpected response.