-
Notifications
You must be signed in to change notification settings - Fork 18
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
http 403 on intermediate files when reading GPKG from S3 #413
Comments
Further, I get something similar when reading raster data using
So, should I really be including a Corollary: do these calls all generally require write access to local filesystems when you read data? |
@gtmaskall this is probably more a question for GDAL (and how it connects with S3 exactly). I also see an initial warning when trying this with the GDAL command line (before failing because I don't have access):
To understand better which requests are being made, you could maybe set |
you may want to set the GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR config option (OSGeo/gdal#9443 (comment) / https://gdal.org/user/configoptions.html#performance-and-caching) to prevent GDAL from issuing a directory listing HTTP request which might not be sufficient here. So you may need to set CPL_VSIL_CURL_ALLOWED_EXTENSIONS to ".gpkg" (cf https://gdal.org/user/configoptions.html#networking-options) |
I'm testing reading vector data from S3.
s3fs
is installed in my environment. I've created a public bucket with a bucket policy granting any principal the s3:GetObject action to objects in the bucket. I'm deliberately avoiding access and secret access keys because the intention is to enable access via roles on the hosting EC2 instance. Thus, I'm specifying:pyogrio
engine does successfully return the data, but with RuntimeWarnings:fiona
also hits a 403, but bombs out and doesn't return data:Is this use case an anti-pattern? Is
pyogrio
looking for these extensions to use them if they exist (and the triggering of the 403 is unfortunate), or is it trying to create them as intermediates? I'm assuming the former, due to the reference toogr_read()
.If it's just a warning, then it's unfortunate but "meh". But should I allow write access so something (GDAL?) can create these intermediates if there's an advantage to
pyogrio
when reading data?The text was updated successfully, but these errors were encountered: