Windows and S2 zip path

[bastiencerr]

Hey, it's me again and i hope for the last time 😄 !

I've a python code which process Sentinel-2 images in the .SAFE format, and it's perfect (thanks again for this library!).
I've tried to make it works with the .zip format in the purpose to avoid manipulation of unzipping the files (it would be long for dozens images).

Unfortunately, when i try, eoreaders crashs.

An exemple of my script would be :

reader = Reader()
s2_path = join(r"C:\Users\BC\Desktop\PW\Data", "S2A_MSIL2A_20220716T103641_N0400_R008_T31TFJ_20220716T183414.zip")
s2_prod = reader.open(s2_path)

This part works well.

It crashs here :

bands = s2_prod.load([NDVI, NDMI, NDWI, UI])

$\textcolor{red}{\textsf{Error message :}}$

CPLE_OpenFailedError                      Traceback (most recent call last)
File rasterio\_base.pyx:310, in rasterio._base.DatasetBase.__init__()

File rasterio\_base.pyx:221, in rasterio._base.open_dataset()

File rasterio\_err.pyx:221, in rasterio._err.exc_wrap_pointer()

CPLE_OpenFailedError: '/vsizip/C:\Users\BC\Desktop\PW\Data\S2A_MSIL2A_20220716T103641_N0400_R008_T31TFJ_20220716T183414.zip!S2A_MSIL2A_20220716T103641_N0400_R008_T31TFJ_20220716T183414.SAFE/GRANULE/L2A_T31TFJ_A036899_20220716T104613/IMG_DATA/R10m/T31TFJ_20220716T103641_B08_10m.jp2' does not exist in the file system, and is not recognized as a supported dataset name.

During handling of the above exception, another exception occurred:

RasterioIOError                           Traceback (most recent call last)
File C:\ProgramData\miniconda3\envs\earth-analytics-python\lib\site-packages\eoreader\utils.py:176, in read(path, pixel_size, size, resampling, masked, indexes, **kwargs)
    175 warnings.simplefilter("ignore", category=NotGeoreferencedWarning)
--> 176 arr = rasters.read(
    177     path,
    178     resolution=pixel_size,
    179     resampling=resampling,
    180     masked=masked,
    181     indexes=indexes,
    182     size=size,
    183     window=window,
    184     chunks=chunks,
    185     **_prune_keywords(
    186         additional_keywords=["window", "chunks", "resolution"], **kwargs
    187     ),
    188 )
    190 # In EOReader, we don't care about the band coordinate of a band loaded from a stack.
    191 # Overwrite it (don't keep the number "2" if we loaded the second band of the stack)

File C:\ProgramData\miniconda3\envs\earth-analytics-python\lib\site-packages\sertit\rasters_rio.py:184, in path_arr_dst.<locals>.path_or_arr_or_dst_wrapper(path_or_arr_or_ds, *args, **kwargs)
    183 if isinstance(path_or_arr_or_ds, (str, Path, CloudPath)):
--> 184     with rasterio.open(str(path_or_arr_or_ds)) as ds:
    185         out = function(ds, *args, **kwargs)

File C:\ProgramData\miniconda3\envs\earth-analytics-python\lib\site-packages\rasterio\env.py:451, in ensure_env_with_credentials.<locals>.wrapper(*args, **kwds)
    450 with env_ctor(session=session):
--> 451     return f(*args, **kwds)

File C:\ProgramData\miniconda3\envs\earth-analytics-python\lib\site-packages\rasterio\__init__.py:304, in open(fp, mode, driver, width, height, count, crs, transform, dtype, nodata, sharing, **kwargs)
    303 if mode == "r":
--> 304     dataset = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)
    305 elif mode == "r+":

File rasterio\_base.pyx:312, in rasterio._base.DatasetBase.__init__()

RasterioIOError: '/vsizip/C:\Users\BC\Desktop\PW\Data\S2A_MSIL2A_20220716T103641_N0400_R008_T31TFJ_20220716T183414.zip!S2A_MSIL2A_20220716T103641_N0400_R008_T31TFJ_20220716T183414.SAFE/GRANULE/L2A_T31TFJ_A036899_20220716T104613/IMG_DATA/R10m/T31TFJ_20220716T103641_B08_10m.jp2' does not exist in the file system, and is not recognized as a supported dataset name.

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
Cell In[7], line 1
----> 1 bands = s2_prod.load([NDVI, NDMI, NDWI, UI])

File C:\ProgramData\miniconda3\envs\earth-analytics-python\lib\site-packages\eoreader\products\product.py:884, in Product.load(self, bands, pixel_size, size, **kwargs)
    882 # Load bands (only once ! and convert the bands to be loaded to correct format)
    883 unique_bands = misc.unique(bands)
--> 884 band_xds = self._load(unique_bands, pixel_size, size, **kwargs)
    886 # Rename all bands and add attributes
    887 for key, val in band_xds.items():

File C:\ProgramData\miniconda3\envs\earth-analytics-python\lib\site-packages\eoreader\products\product.py:982, in Product._load(self, bands, pixel_size, size, **kwargs)
    980 if unique_bands:
    981     LOGGER.debug(f"Loading bands {to_str(unique_bands)}")
--> 982     loaded_bands = self._load_bands(
    983         unique_bands, pixel_size=pixel_size, size=size, **kwargs
    984     )
    986     # Compute index (they conserve the nodata)
    987     if index_list:
    988         # Collocate bands before indices to ensure the same size to perform operations between bands

File C:\ProgramData\miniconda3\envs\earth-analytics-python\lib\site-packages\eoreader\products\optical\s2_product.py:1185, in S2Product._load_bands(self, bands, pixel_size, size, **kwargs)
   1182 band_paths = self.get_band_paths(bands, pixel_size=pixel_size, **kwargs)
   1184 # Open bands and get array (resampled if needed)
-> 1185 band_arrays = self._open_bands(
   1186     band_paths, pixel_size=pixel_size, size=size, **kwargs
   1187 )
   1189 return band_arrays

File C:\ProgramData\miniconda3\envs\earth-analytics-python\lib\site-packages\eoreader\products\optical\optical_product.py:315, in OpticalProduct._open_bands(self, band_paths, pixel_size, size, **kwargs)
    312 for band, band_path in band_paths.items():
    313     # Read band
    314     LOGGER.debug(f"Read {band.name}")
--> 315     band_arr = self._read_band(
    316         band_path, band=band, pixel_size=pixel_size, size=size, **kwargs
    317     )
    319     if not pixel_size:
    320         pixel_size = band_arr.rio.resolution()[0]

File C:\ProgramData\miniconda3\envs\earth-analytics-python\lib\site-packages\eoreader\products\optical\s2_product.py:690, in S2Product._read_band(self, path, band, pixel_size, size, **kwargs)
    687         raise ex
    689 # Read band
--> 690 return utils.read(
    691     geocoded_path,
    692     pixel_size=pixel_size,
    693     size=size,
    694     resampling=Resampling.bilinear,
    695     **kwargs,
    696 )

File C:\ProgramData\miniconda3\envs\earth-analytics-python\lib\site-packages\eoreader\utils.py:201, in read(path, pixel_size, size, resampling, masked, indexes, **kwargs)
    198         return arr
    200 except errors.RasterioIOError as ex:
--> 201     if (str(path).endswith("jp2") or str(path).endswith("tif")) and path.exists():
    202         raise InvalidProductError(f"Corrupted file: {path}") from ex
    203     else:

AttributeError: 'str' object has no attribute 'exists'

I've also tried by putting my Sentinel-2 path as a Path object (with pathlib library) but i had the same issue.

Do you have any idea ?

[remi-braun]

This is a bug !
I'll correct you this and do a new release asap 😅

[bastiencerr]

Alright ! I will track the release then ! Thanks !

[remi-braun]

This is only something that seems to happen on Windows. Could you just test it on Linux if you can ? This would explain why my tests are passing fine.

[bastiencerr]

I do not have access to any linux machine, i can try to download a vm if you want

[remi-braun]

No it's OK, I managed to reproduce it so it's fine 😉

[remi-braun]

I just released sertit 1.28.3 that should correct your issue.
In a few minutes, the wheel will be available on pypi, so you will be able to do pip install sertit[full]==1.28.3 to update your env :)

[bastiencerr]

It works !

Thanks for the fix and the fast answer !