How to use rasterio read gdal dataset created in memory?

[sqwev]

How can I use rasterio read gdal dataset created in memory?

for example:

mem_driver = gdal.GetDriverByName('MEM')
# get source epsg
ds = mem_driver.Create('', WIDTH, HEIGHT, BANDS, gdal_type)
ds.SetProjection(projection)
ds.SetGeoTransform(geoTransform)

I have try to use memory file path in gdal, but it's failed.

vsi_mem_file_path = f'/vsimem/{uuid.uuid4().hex}.tif'
gdal.GetDriverByName('GTiff').CreateCopy(vsi_mem_file_path, ds)

with rasterio.open(vsi_mem_file_path ) as ref_src:

How can I use rasterio to read ds object, so I can use some functions implemented in rasterio but not in gdal?

Thank you for your concern and help.

[sgillies]

@sqwev GDAL's MEM format and /vsimem/ are two different things. MEM format datasets exist in different memory from /vsimem/ files, so yes, you would have to copy. I would expect your code to work if gdal.Open() successfully opens the /vsimem/ file. Can you share the error message you see?

[sqwev]

Sorry, I need to provide more detailed information.

I execute this code on Windows 11, my GDAL version is 3.7.1, rasterio version is 1.3.9, python version is 3.9.17

# create gdal dataset in memory
mem_driver = gdal.GetDriverByName('MEM')
ds = mem_driver.Create('', WIDTH, HEIGHT, BANDS, gdal_type)
ds.SetProjection(projection)
ds.SetGeoTransform(geoTransform)

# create 
vsi_mem_file_path = f'/vsimem/{uuid.uuid4().hex}.tif'
gdal.GetDriverByName('GTiff').CreateCopy(vsi_mem_file_path, ds)

# test if gdal can open
print(f"vsi_mem_file_path : {vsi_mem_file_path }")
test_ds = gdal.Open(vsi_mem_file_path )
print(f"test_ds: {test_ds}")
print(f"test_ds.RasterCount: {test_ds.RasterCount}")
print(f"test_ds.RasterXSize: {test_ds.RasterXSize}")
print(f"test_ds.RasterYSize: {test_ds.RasterYSize}")
with rasterio.open(vsi_mem_file_path) as ref_src:
    print(f"ref_src: {ref_src}")

The results are as follows:

vsi_mem_file_path : /vsimem/e9f6eb94a8fe4d529244de625027973f.tif
test_ds: <osgeo.gdal.Dataset; proxy of <Swig Object of type 'GDALDatasetShadow *' at 0x000002723929E240> >
test_ds.RasterCount: 3
test_ds.RasterXSize: 5255
test_ds.RasterYSize: 2410
Traceback (most recent call last):
  File "rasterio\\_base.pyx", line 310, in rasterio._base.DatasetBase.__init__
  File "rasterio\\_base.pyx", line 221, in rasterio._base.open_dataset
  File "rasterio\\_err.pyx", line 221, in rasterio._err.exc_wrap_pointer
rasterio._err.CPLE_OpenFailedError: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\phd\CropMaster\test\get_ds_for_paper.py", line 159, in <module>
    cluster_yield_ds = fo.yield_mask(yield_df, True, False, cluster_yield_path)
  File "D:\phd\CropMaster\cropbase\farm\farm.py", line 1083, in yield_mask
    yield_img = self.process_by_field_img(process_fun=gen_augyield_by_field,
  File "D:\phd\CropMaster\cropbase\farm\farm.py", line 719, in process_by_field_img
    with rasterio.open(vsi_mem_file_path) as ref_src:
  File "C:\Users\shenzhou\anaconda3\envs\rs\lib\site-packages\rasterio\env.py", line 451, in wrapper
    return f(*args, **kwds)
  File "C:\Users\shenzhou\anaconda3\envs\rs\lib\site-packages\rasterio\__init__.py", line 331, in open
    dataset = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)
  File "rasterio\\_base.pyx", line 312, in rasterio._base.DatasetBase.__init__
rasterio.errors.RasterioIOError: No such file or directory

It seems that gdal.Open() works well, but with rasterio.open() failed. Thank you for your assistance.

[sgillies]

I ran this on my macbook. osgeo.gdal and rasterio are installed from conda-forge, the latest of each. If you installed from a different conda channel, it would be good to reinstall from conda-forge.

import uuid
from osgeo import gdal
import rasterio

# create gdal dataset in memory
mem_driver = gdal.GetDriverByName('MEM')
ds = mem_driver.Create('', 16, 16, 1, gdal.GDT_Byte)
# ds.SetProjection(projection)
# ds.SetGeoTransform(geoTransform)

# create 
vsi_mem_file_path = f'/vsimem/{uuid.uuid4().hex}.tif'
gdal.GetDriverByName('GTiff').CreateCopy(vsi_mem_file_path, ds)

# test if gdal can open
print(f"vsi_mem_file_path : {vsi_mem_file_path }")
test_ds = gdal.Open(vsi_mem_file_path )
print(f"test_ds: {test_ds}")
print(f"test_ds.RasterCount: {test_ds.RasterCount}")
print(f"test_ds.RasterXSize: {test_ds.RasterXSize}")
print(f"test_ds.RasterYSize: {test_ds.RasterYSize}")
with rasterio.open(vsi_mem_file_path) as ref_src:
    print(f"ref_src: {ref_src}")

The output:

/Users/sean.gillies/miniforge3/envs/first/lib/python3.11/site-packages/osgeo/gdal.py:312: FutureWarning: Neither gdal.UseExceptions() nor gdal.DontUseExceptions() has been explicitly called. In GDAL 4.0, exceptions will be enabled by default.
  warnings.warn(
vsi_mem_file_path : /vsimem/852bd80ed60440e388cb13e3e35ce068.tif
test_ds: <osgeo.gdal.Dataset; proxy of <Swig Object of type 'GDALDatasetShadow *' at 0x10569d680> >
test_ds.RasterCount: 1
test_ds.RasterXSize: 16
test_ds.RasterYSize: 16
/Users/sean.gillies/miniforge3/envs/first/lib/python3.11/site-packages/rasterio/__init__.py:304: NotGeoreferencedWarning: Dataset has no geotransform, gcps, or rpcs. The identity matrix will be returned.
  dataset = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)
ref_src: <open DatasetReader name='/vsimem/852bd80ed60440e388cb13e3e35ce068.tif' mode='r'>

It should work, unless there is a problem specific to Windows. That seems unlikely to me. Rasterio isn't doing anything OS specific when it comes to opening a dataset.

[sqwev]

I follow your execution and it works well, the GDAL in my previous environment is installed from wheel file, and rasterio is installed from pip. Thank you for your help!