Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate higher HEAD/GET Request with GDAL >3.7 #697

Open
vincentsarago opened this issue Apr 18, 2024 · 9 comments
Open

Investigate higher HEAD/GET Request with GDAL >3.7 #697

vincentsarago opened this issue Apr 18, 2024 · 9 comments
Labels
bug

Comments

@vincentsarago
Copy link
Member

with https://github.com/cogeotiff/rio-tiler/blob/main/tests/benchmarks/requests.py we test some tile request fetching and use tilebench to monitor the number or GET/HEAD requests made.

with the latest commits, this test is failing

rio-tiler/tests/benchmarks/requests.py

Lines 35 to 67 in 4d95c28

def test_tile_read():
"""Tile Read tests."""
@profile(
kernels=True,
config={
"GDAL_HTTP_MERGE_CONSECUTIVE_RANGES": "YES",
"GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR",
"GDAL_INGESTED_BYTES_AT_OPEN": 32768,
"CPL_VSIL_CURL_NON_CACHED": f"/vsicurl/{dataset_url}",
},
quiet=True,
add_to_return=True,
)
def tile(src_path: str, x: int, y: int, z: int):
with Reader(src_path) as src_dst:
return src_dst.tile(x, y, z, tilesize=256)
_, stats = tile(
dataset_url,
493,
741,
11,
)
assert stats["HEAD"]["count"] == 1
assert stats["GET"]["count"] == 3
assert stats["GET"]["bytes"] == 2293760
assert stats["GET"]["ranges"] == [
"0-32767",
"9011200-10158079",
"11075584-12189695",
]
assert stats["WarpKernels"]

👇 this used to return HEAD: 1 and GET 3 🤷

tilebench profile https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif --tile 11-493-741 | jq
{
  "HEAD": {
    "count": 3
  },
  "GET": {
    "count": 4,
    "bytes": 2293760,
    "ranges": [
      "0-32767",
      "9011200-10158079",
      "11075584-12189695"
    ]
  },
  "Timing": 3.494839668273926
}
@vincentsarago
Copy link
Member Author

vincentsarago commented Apr 18, 2024 •
edited

Rasterio 1.3.9 / GDAL 3.6.4

tilebench profile https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif --tile 11-493-741 | jq
{
  "HEAD": {
    "count": 1
  },
  "GET": {
    "count": 3,
    "bytes": 2293760,
    "ranges": [
      "0-32767",
      "9011200-10158079",
      "11075584-12189695"
    ]
  },
  "Timing": 2.6649599075317383
}

Rasterio 1.3.10 / GDAL 3.7.3

{
  "HEAD": {
    "count": 3
  },
  "GET": {
    "count": 4,
    "bytes": 2293760,
    "ranges": [
      "0-32767",
      "9011200-10158079",
      "11075584-12189695"
    ]
  },
  "Timing": 3.2888236045837402
}

Rasterio 1.3.10 / GDAL 3.8.4

 tilebench profile https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif --tile 11-493-741 | jq
{
  "HEAD": {
    "count": 3
  },
  "GET": {
    "count": 4,
    "bytes": 2293760,
    "ranges": [
      "0-32767",
      "9011200-10158079",
      "11075584-12189695"
    ]
  },
  "Timing": 2.7414448261260986
}

CURL Logs

tilebench profile https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif --tile 11-493-741 --add-stdout | jq -r '.logs[]' | grep "CURL_INFO_HEADER_OUT"
CPLE_None in CURL_INFO_HEADER_OUT: HEAD /sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif HTTP/1.1
CPLE_None in CURL_INFO_HEADER_OUT: GET /sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif HTTP/1.1
CPLE_None in CURL_INFO_HEADER_OUT: GET /sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/ HTTP/1.1
CPLE_None in CURL_INFO_HEADER_OUT: HEAD /sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.xml HTTP/1.1
CPLE_None in CURL_INFO_HEADER_OUT: HEAD /sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.XML HTTP/1.1
CPLE_None in CURL_INFO_HEADER_OUT: GET /sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif HTTP/1.1
CPLE_None in CURL_INFO_HEADER_OUT: GET /sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif HTTP/1.1

tilebench profile https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif --tile 11-493-741 --add-stdout | jq '.logs'
[
  "Entering env context: <rasterio.env.Env object at 0x13fa59460>",
  "Starting outermost env",
  "No GDAL environment exists",
  "New GDAL environment <rasterio._env.GDALEnv object at 0x13fa599d0> created",
  "Installing FilePath filesystem handler plugin...",
  "GDAL_DATA found in environment.",
  "PROJ_LIB found in environment.",
  "Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x13fa599d0>.",
  "Entered env context: <rasterio.env.Env object at 0x13fa59460>",
  "Got a copy of environment <rasterio._env.GDALEnv object at 0x13fa599d0> options",
  "Entering env context: <rasterio.env.Env object at 0x13fad4820>",
  "Got a copy of environment <rasterio._env.GDALEnv object at 0x13fa599d0> options",
  "Entered env context: <rasterio.env.Env object at 0x13fad4820>",
  "Sharing flag: 0",
  "CPLE_None in HTTP: libcurl/8.4.0 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.12 nghttp2/1.58.0",
  "CPLE_None in HTTP: GDAL was built against curl 7.87.0, but is running against 8.4.0.",
  "CPLE_None in CURL_INFO_TEXT: Couldn't find host sentinel-cogs.s3.us-west-2.amazonaws.com in the .netrc file; using defaults",
  "CPLE_None in CURL_INFO_TEXT:   Trying 52.218.232.177:443...",
  "CPLE_None in CURL_INFO_TEXT: Connected to sentinel-cogs.s3.us-west-2.amazonaws.com (52.218.232.177) port 443",
  "CPLE_None in CURL_INFO_TEXT: ALPN: curl offers h2,http/1.1",
  "CPLE_None in CURL_INFO_TEXT: (304) (OUT), TLS handshake, Client hello (1):",
  "CPLE_None in CURL_INFO_TEXT:  CAfile: /Users/vincentsarago/Dev/venv/py39/lib/python3.9/site-packages/certifi/cacert.pem",
  "CPLE_None in CURL_INFO_TEXT:  CApath: none",
  "CPLE_None in CURL_INFO_TEXT: (304) (IN), TLS handshake, Server hello (2):",
  "CPLE_None in CURL_INFO_TEXT: TLSv1.2 (IN), TLS handshake, Certificate (11):",
  "CPLE_None in CURL_INFO_TEXT: TLSv1.2 (IN), TLS handshake, Server key exchange (12):",
  "CPLE_None in CURL_INFO_TEXT: TLSv1.2 (IN), TLS handshake, Server finished (14):",
  "CPLE_None in CURL_INFO_TEXT: TLSv1.2 (OUT), TLS handshake, Client key exchange (16):",
  "CPLE_None in CURL_INFO_TEXT: TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):",
  "CPLE_None in CURL_INFO_TEXT: TLSv1.2 (OUT), TLS handshake, Finished (20):",
  "CPLE_None in CURL_INFO_TEXT: TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):",
  "CPLE_None in CURL_INFO_TEXT: TLSv1.2 (IN), TLS handshake, Finished (20):",
  "CPLE_None in CURL_INFO_TEXT: SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256",
  "CPLE_None in CURL_INFO_TEXT: ALPN: server accepted http/1.1",
  "CPLE_None in CURL_INFO_TEXT: Server certificate:",
  "CPLE_None in CURL_INFO_TEXT:  subject: CN=*.s3-us-west-2.amazonaws.com",
  "CPLE_None in CURL_INFO_TEXT:  start date: Mar 15 00:00:00 2024 GMT",
  "CPLE_None in CURL_INFO_TEXT:  expire date: Feb 15 23:59:59 2025 GMT",
  "CPLE_None in CURL_INFO_TEXT:  subjectAltName: host \"sentinel-cogs.s3.us-west-2.amazonaws.com\" matched cert's \"*.s3.us-west-2.amazonaws.com\"",
  "CPLE_None in CURL_INFO_TEXT:  issuer: C=US; O=Amazon; CN=Amazon RSA 2048 M01",
  "CPLE_None in CURL_INFO_TEXT:  SSL certificate verify ok.",
  "CPLE_None in CURL_INFO_TEXT: using HTTP/1.1",
  "CPLE_None in CURL_INFO_HEADER_OUT: HEAD /sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif HTTP/1.1",
  "Host: sentinel-cogs.s3.us-west-2.amazonaws.com",
  "User-Agent: GDAL/3.7.3",
  "Accept: */*",
  "",
  "CPLE_None in CURL_INFO_HEADER_IN: HTTP/1.1 200 OK",
  "CPLE_None in CURL_INFO_HEADER_IN: x-amz-id-2: glv7Xi1AW/YS7UJsgsQAoYokXq/gB35c8LMEJYo2FLKpI+o1tR6xsyfWKou6TgrfZgcuY2ooDYQ=",
  "CPLE_None in CURL_INFO_HEADER_IN: x-amz-request-id: 2VMQ56MHK9DHVN5M",
  "CPLE_None in CURL_INFO_HEADER_IN: Date: Thu, 18 Apr 2024 13:54:32 GMT",
  "CPLE_None in CURL_INFO_HEADER_IN: Last-Modified: Mon, 09 Oct 2023 01:23:09 GMT",
  "CPLE_None in CURL_INFO_HEADER_IN: ETag: \"3e23880cd7ac8a32ae71c55ad4f85c5d-24\"",
  "CPLE_None in CURL_INFO_HEADER_IN: x-amz-storage-class: INTELLIGENT_TIERING",
  "CPLE_None in CURL_INFO_HEADER_IN: x-amz-server-side-encryption: AES256",
  "CPLE_None in CURL_INFO_HEADER_IN: Cache-Control: public, max-age=31536000, immutable",
  "CPLE_None in CURL_INFO_HEADER_IN: Accept-Ranges: bytes",
  "CPLE_None in CURL_INFO_HEADER_IN: Content-Type: image/tiff; application=geotiff; profile=cloud-optimized",
  "CPLE_None in CURL_INFO_HEADER_IN: Server: AmazonS3",
  "CPLE_None in CURL_INFO_HEADER_IN: Content-Length: 200959614",
  "CPLE_None in CURL_INFO_HEADER_IN: ",
  "CPLE_None in CURL_INFO_TEXT: Connection #0 to host sentinel-cogs.s3.us-west-2.amazonaws.com left intact",
  "CPLE_None in VSICURL: GetFileSize(https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif)=200959614  response_code=200",
  "CPLE_None in VSICURL: Downloading 0-32767 (https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif)...",
  "CPLE_None in CURL_INFO_TEXT: Couldn't find host sentinel-cogs.s3.us-west-2.amazonaws.com in the .netrc file; using defaults",
  "CPLE_None in CURL_INFO_TEXT: Found bundle for host: 0x13f8f81e0 [serially]",
  "CPLE_None in CURL_INFO_TEXT: Can not multiplex, even if we wanted to",
  "CPLE_None in CURL_INFO_TEXT: Re-using existing connection with host sentinel-cogs.s3.us-west-2.amazonaws.com",
  "CPLE_None in CURL_INFO_HEADER_OUT: GET /sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif HTTP/1.1",
  "Host: sentinel-cogs.s3.us-west-2.amazonaws.com",
  "User-Agent: GDAL/3.7.3",
  "Accept: */*",
  "Range: bytes=0-32767",
  "",
  "CPLE_None in CURL_INFO_HEADER_IN: HTTP/1.1 206 Partial Content",
  "CPLE_None in CURL_INFO_HEADER_IN: x-amz-id-2: pTwUP0oEk8NsXKYMEHOeoWlAF5VlM6s/F4HLQPcdYMBbkPD13uqFkzOzdrCLv/JkFVg1LMJdeIc=",
  "CPLE_None in CURL_INFO_HEADER_IN: x-amz-request-id: 2VMYTCRG7WAEDXPJ",
  "CPLE_None in CURL_INFO_HEADER_IN: Date: Thu, 18 Apr 2024 13:54:32 GMT",
  "CPLE_None in CURL_INFO_HEADER_IN: Last-Modified: Mon, 09 Oct 2023 01:23:09 GMT",
  "CPLE_None in CURL_INFO_HEADER_IN: ETag: \"3e23880cd7ac8a32ae71c55ad4f85c5d-24\"",
  "CPLE_None in CURL_INFO_HEADER_IN: x-amz-storage-class: INTELLIGENT_TIERING",
  "CPLE_None in CURL_INFO_HEADER_IN: x-amz-server-side-encryption: AES256",
  "CPLE_None in CURL_INFO_HEADER_IN: Cache-Control: public, max-age=31536000, immutable",
  "CPLE_None in CURL_INFO_HEADER_IN: Accept-Ranges: bytes",
  "CPLE_None in CURL_INFO_HEADER_IN: Content-Range: bytes 0-32767/200959614",
  "CPLE_None in CURL_INFO_HEADER_IN: Content-Type: image/tiff; application=geotiff; profile=cloud-optimized",
  "CPLE_None in CURL_INFO_HEADER_IN: Server: AmazonS3",
  "CPLE_None in CURL_INFO_HEADER_IN: Content-Length: 32768",
  "CPLE_None in CURL_INFO_HEADER_IN: ",
  "CPLE_None in CURL_INFO_TEXT: Connection #0 to host sentinel-cogs.s3.us-west-2.amazonaws.com left intact",
  "CPLE_None in VSICURL: Got response_code=206",
  "CPLE_None in GDAL: GDALOpen(/vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif, this=0x11f249ca0) succeeds as GTiff.",
  "Nodata success: 1, Nodata value: 0.000000",
  "Nodata success: 1, Nodata value: 0.000000",
  "Nodata success: 1, Nodata value: 0.000000",
  "Dataset <open DatasetReader name='https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif' mode='r'> is started.",
  "Exiting env context: <rasterio.env.Env object at 0x13fad4820>",
  "Cleared existing <rasterio._env.GDALEnv object at 0x13fa599d0> options",
  "Stopped GDALEnv <rasterio._env.GDALEnv object at 0x13fa599d0>.",
  "No GDAL environment exists",
  "New GDAL environment <rasterio._env.GDALEnv object at 0x13fa599d0> created",
  "GDAL_DATA found in environment.",
  "PROJ_LIB found in environment.",
  "Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x13fa599d0>.",
  "Exited env context: <rasterio.env.Env object at 0x13fad4820>",
  "CPLE_None in GTiff: ScanDirectories()",
  "CPLE_None in GTiff: Opened 5490x5490 overview.",
  "CPLE_None in GTiff: Opened 2745x2745 overview.",
  "CPLE_None in GTiff: Opened 1373x1373 overview.",
  "CPLE_None in GTiff: Opened 687x687 overview.",
  "Matched. confidence=100, c_code=b'3857', c_name=b'EPSG'",
  "Matched. confidence=100, c_code=b'32615', c_name=b'EPSG'",
  "CPLE_None in VRT: No valid sources found for band in VRT file ",
  "CPLE_None in GDAL: GDALOpen(<VRTDataset rasterYSize=\"10980\" rasterXSize=\"10980\"><VRTRasterBand /><SRS>PROJCS[\"WGS 84 / UTM zone 15N\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-93],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"EPSG\",\"32615\"]]</SRS><GeoTransform>399960.0,10.0,0.0,5000040.0,0.0,-10.0</GeoTransform></VRTDataset>, this=0x13fc0a1f0) succeeds as VRT.",
  "CPLE_None in GDAL: Computing area of interest: -94.2725, 44.1585, -92.8758, 45.1538",
  "Created exact transformer",
  "CPLE_None in GDAL: GDALClose(<VRTDataset rasterYSize=\"10980\" rasterXSize=\"10980\"><VRTRasterBand /><SRS>PROJCS[\"WGS 84 / UTM zone 15N\",GEOGCS[\"WGS 84\",DATUM[\"WGS_1984\",SPHEROID[\"WGS 84\",6378137,298.257223563,AUTHORITY[\"EPSG\",\"7030\"]],AUTHORITY[\"EPSG\",\"6326\"]],PRIMEM[\"Greenwich\",0,AUTHORITY[\"EPSG\",\"8901\"]],UNIT[\"degree\",0.0174532925199433,AUTHORITY[\"EPSG\",\"9122\"]],AUTHORITY[\"EPSG\",\"4326\"]],PROJECTION[\"Transverse_Mercator\"],PARAMETER[\"latitude_of_origin\",0],PARAMETER[\"central_meridian\",-93],PARAMETER[\"scale_factor\",0.9996],PARAMETER[\"false_easting\",500000],PARAMETER[\"false_northing\",0],UNIT[\"metre\",1,AUTHORITY[\"EPSG\",\"9001\"]],AXIS[\"Easting\",EAST],AXIS[\"Northing\",NORTH],AUTHORITY[\"EPSG\",\"32615\"]]</SRS><GeoTransform>399960.0,10.0,0.0,5000040.0,0.0,-10.0</GeoTransform></VRTDataset>, this=0x13fc0a1f0)",
  "Exported CRS to WKT.",
  "Warp_extras: {'init_dest': 'NO_DATA'}",
  "Nodata success: 1, Nodata value: 0.000000",
  "Nodata success: 1, Nodata value: 0.000000",
  "Nodata success: 1, Nodata value: 0.000000",
  "Dataset <closed WarpedVRT name='WarpedVRT(https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif)' mode='r'> is started.",
  "Output nodata value read from file: 0.0",
  "Output nodata value read from file: 0.0",
  "Output nodata value read from file: 0.0",
  "Output nodata values: [0.0, 0.0, 0.0]",
  "all_valid: False",
  "mask_flags: ([<MaskFlags.nodata: 8>], [<MaskFlags.nodata: 8>], [<MaskFlags.nodata: 8>])",
  "Jump straight to _read()",
  "Window: None",
  "IO window xoff=0.0 yoff=0.0 width=1391.0 height=1391.0",
  "CPLE_None in GDAL: GDALOverviewDataset(/vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif, this=0x13fc0a1f0) creation.",
  "CPLE_None in VSICURL: GetFileList(/vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A)",
  "CPLE_None in CURL_INFO_TEXT: Couldn't find host sentinel-cogs.s3.us-west-2.amazonaws.com in the .netrc file; using defaults",
  "CPLE_None in CURL_INFO_TEXT: Found bundle for host: 0x13f8f81e0 [serially]",
  "CPLE_None in CURL_INFO_TEXT: Can not multiplex, even if we wanted to",
  "CPLE_None in CURL_INFO_TEXT: Re-using existing connection with host sentinel-cogs.s3.us-west-2.amazonaws.com",
  "CPLE_None in CURL_INFO_HEADER_OUT: GET /sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/ HTTP/1.1",
  "Host: sentinel-cogs.s3.us-west-2.amazonaws.com",
  "User-Agent: GDAL/3.7.3",
  "Accept: */*",
  "",
  "CPLE_None in CURL_INFO_HEADER_IN: HTTP/1.1 404 Not Found",
  "CPLE_None in CURL_INFO_HEADER_IN: x-amz-request-id: EJXKH2VG0DDEA2TP",
  "CPLE_None in CURL_INFO_HEADER_IN: x-amz-id-2: 9ZbkWtfMg77R0o/nF4+/8n2OmcPdpK0ibQYNx5hoUXfPkKO3L4KlHK5qiimT/t5SmuSFoajsV5k=",
  "CPLE_None in CURL_INFO_HEADER_IN: Content-Type: application/xml",
  "CPLE_None in CURL_INFO_HEADER_IN: Transfer-Encoding: chunked",
  "CPLE_None in CURL_INFO_HEADER_IN: Date: Thu, 18 Apr 2024 13:54:31 GMT",
  "CPLE_None in CURL_INFO_HEADER_IN: Server: AmazonS3",
  "CPLE_None in CURL_INFO_HEADER_IN: ",
  "CPLE_None in CURL_INFO_TEXT: Connection #0 to host sentinel-cogs.s3.us-west-2.amazonaws.com left intact",
  "CPLE_None in CURL_INFO_TEXT: Couldn't find host sentinel-cogs.s3.us-west-2.amazonaws.com in the .netrc file; using defaults",
  "CPLE_None in CURL_INFO_TEXT: Found bundle for host: 0x13f8f81e0 [serially]",
  "CPLE_None in CURL_INFO_TEXT: Can not multiplex, even if we wanted to",
  "CPLE_None in CURL_INFO_TEXT: Re-using existing connection with host sentinel-cogs.s3.us-west-2.amazonaws.com",
  "CPLE_None in CURL_INFO_HEADER_OUT: HEAD /sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.xml HTTP/1.1",
  "Host: sentinel-cogs.s3.us-west-2.amazonaws.com",
  "User-Agent: GDAL/3.7.3",
  "Accept: */*",
  "",
  "CPLE_None in CURL_INFO_HEADER_IN: HTTP/1.1 404 Not Found",
  "CPLE_None in CURL_INFO_HEADER_IN: x-amz-request-id: EJXNYD4M1D6W2XSH",
  "CPLE_None in CURL_INFO_HEADER_IN: x-amz-id-2: ww4bqRaqhxyjtj+DTfhePvtfVUJnIYdVC8HlpSTobCbJFGRomigMMM06NOTZZnCZ2DC8ntOMK1Y=",
  "CPLE_None in CURL_INFO_HEADER_IN: Content-Type: application/xml",
  "CPLE_None in CURL_INFO_HEADER_IN: Date: Thu, 18 Apr 2024 13:54:31 GMT",
  "CPLE_None in CURL_INFO_HEADER_IN: Server: AmazonS3",
  "CPLE_None in CURL_INFO_HEADER_IN: ",
  "CPLE_None in CURL_INFO_TEXT: Connection #0 to host sentinel-cogs.s3.us-west-2.amazonaws.com left intact",
  "CPLE_None in VSICURL: GetFileSize(https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.xml): response_code=404, server error msg=HTTP/1.1 404 Not Found",
  "x-amz-request-id: EJXNYD4M1D6W2XSH",
  "x-amz-id-2: ww4bqRaqhxyjtj+DTfhePvtfVUJnIYdVC8HlpSTobCbJFGRomigMMM06NOTZZnCZ2DC8ntOMK1Y=",
  "Content-Type: application/xml",
  "Date: Thu, 18 Apr 2024 13:54:31 GMT",
  "Server: AmazonS3",
  "",
  "",
  "CPLE_None in CURL_INFO_TEXT: Couldn't find host sentinel-cogs.s3.us-west-2.amazonaws.com in the .netrc file; using defaults",
  "CPLE_None in CURL_INFO_TEXT: Found bundle for host: 0x13f8f81e0 [serially]",
  "CPLE_None in CURL_INFO_TEXT: Can not multiplex, even if we wanted to",
  "CPLE_None in CURL_INFO_TEXT: Re-using existing connection with host sentinel-cogs.s3.us-west-2.amazonaws.com",
  "CPLE_None in CURL_INFO_HEADER_OUT: HEAD /sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.XML HTTP/1.1",
  "Host: sentinel-cogs.s3.us-west-2.amazonaws.com",
  "User-Agent: GDAL/3.7.3",
  "Accept: */*",
  "",
  "CPLE_None in CURL_INFO_HEADER_IN: HTTP/1.1 404 Not Found",
  "CPLE_None in CURL_INFO_HEADER_IN: x-amz-request-id: EJXQCKA8MA3349AW",
  "CPLE_None in CURL_INFO_HEADER_IN: x-amz-id-2: Lbj0y6wdZs3xlpQjojuE0A7vFO/skRqxCSFwJRMcbFT7Uj8jRRp0D+hCtYazzSXB2Rax1RghP7k=",
  "CPLE_None in CURL_INFO_HEADER_IN: Content-Type: application/xml",
  "CPLE_None in CURL_INFO_HEADER_IN: Date: Thu, 18 Apr 2024 13:54:31 GMT",
  "CPLE_None in CURL_INFO_HEADER_IN: Server: AmazonS3",
  "CPLE_None in CURL_INFO_HEADER_IN: ",
  "CPLE_None in CURL_INFO_TEXT: Connection #0 to host sentinel-cogs.s3.us-west-2.amazonaws.com left intact",
  "CPLE_None in VSICURL: GetFileSize(https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.XML): response_code=404, server error msg=HTTP/1.1 404 Not Found",
  "x-amz-request-id: EJXQCKA8MA3349AW",
  "x-amz-id-2: Lbj0y6wdZs3xlpQjojuE0A7vFO/skRqxCSFwJRMcbFT7Uj8jRRp0D+hCtYazzSXB2Rax1RghP7k=",
  "Content-Type: application/xml",
  "Date: Thu, 18 Apr 2024 13:54:31 GMT",
  "Server: AmazonS3",
  "",
  "",
  "CPLE_None in GDAL: GDALOverviewDataset(/vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif, this=0x11f416860) creation.",
  "CPLE_None in GDAL: GDALOverviewDataset(/vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif, this=0x13fc349c0) creation.",
  "CPLE_None in GDAL: GDALOverviewDataset(/vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif, this=0x13fc35350) creation.",
  "CPLE_None in GDAL: GDALOverviewDataset(/vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif, this=0x13fc48aa0) creation.",
  "CPLE_None in GDAL: GDALOverviewDataset(/vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif, this=0x13fc534c0) creation.",
  "CPLE_None in GDAL: GDAL_CACHEMAX = 12288 MB",
  "CPLE_None in VSICURL: Downloading 9011200-10158079 (https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif)...",
  "CPLE_None in CURL_INFO_TEXT: Couldn't find host sentinel-cogs.s3.us-west-2.amazonaws.com in the .netrc file; using defaults",
  "CPLE_None in CURL_INFO_TEXT: Found bundle for host: 0x13f8f81e0 [serially]",
  "CPLE_None in CURL_INFO_TEXT: Can not multiplex, even if we wanted to",
  "CPLE_None in CURL_INFO_TEXT: Re-using existing connection with host sentinel-cogs.s3.us-west-2.amazonaws.com",
  "CPLE_None in CURL_INFO_HEADER_OUT: GET /sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif HTTP/1.1",
  "Host: sentinel-cogs.s3.us-west-2.amazonaws.com",
  "User-Agent: GDAL/3.7.3",
  "Accept: */*",
  "Range: bytes=9011200-10158079",
  "",
  "CPLE_None in CURL_INFO_HEADER_IN: HTTP/1.1 206 Partial Content",
  "CPLE_None in CURL_INFO_HEADER_IN: x-amz-id-2: +NsF6qFLnSt0sbQ9WvuJl3iuifVS2Ggbah7FMHrObNpF9hc3IDG7IJPjkuFm8VlBdq8/m4pEpHE=",
  "CPLE_None in CURL_INFO_HEADER_IN: x-amz-request-id: EJXS85WYC3GDRM76",
  "CPLE_None in CURL_INFO_HEADER_IN: Date: Thu, 18 Apr 2024 13:54:33 GMT",
  "CPLE_None in CURL_INFO_HEADER_IN: Last-Modified: Mon, 09 Oct 2023 01:23:09 GMT",
  "CPLE_None in CURL_INFO_HEADER_IN: ETag: \"3e23880cd7ac8a32ae71c55ad4f85c5d-24\"",
  "CPLE_None in CURL_INFO_HEADER_IN: x-amz-storage-class: INTELLIGENT_TIERING",
  "CPLE_None in CURL_INFO_HEADER_IN: x-amz-server-side-encryption: AES256",
  "CPLE_None in CURL_INFO_HEADER_IN: Cache-Control: public, max-age=31536000, immutable",
  "CPLE_None in CURL_INFO_HEADER_IN: Accept-Ranges: bytes",
  "CPLE_None in CURL_INFO_HEADER_IN: Content-Range: bytes 9011200-10158079/200959614",
  "CPLE_None in CURL_INFO_HEADER_IN: Content-Type: image/tiff; application=geotiff; profile=cloud-optimized",
  "CPLE_None in CURL_INFO_HEADER_IN: Server: AmazonS3",
  "CPLE_None in CURL_INFO_HEADER_IN: Content-Length: 1146880",
  "CPLE_None in CURL_INFO_HEADER_IN: ",
  "CPLE_None in CURL_INFO_TEXT: Connection #0 to host sentinel-cogs.s3.us-west-2.amazonaws.com left intact",
  "CPLE_None in VSICURL: Got response_code=206",
  "CPLE_None in GDAL: GDALWarpKernel()::GWKNearestByte() Src=1824,1911,351x130 Dst=0,0,348x128",
  "CPLE_None in VSICURL: Downloading 11075584-12189695 (https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif)...",
  "CPLE_None in CURL_INFO_TEXT: Couldn't find host sentinel-cogs.s3.us-west-2.amazonaws.com in the .netrc file; using defaults",
  "CPLE_None in CURL_INFO_TEXT: Found bundle for host: 0x13f8f81e0 [serially]",
  "CPLE_None in CURL_INFO_TEXT: Can not multiplex, even if we wanted to",
  "CPLE_None in CURL_INFO_TEXT: Re-using existing connection with host sentinel-cogs.s3.us-west-2.amazonaws.com",
  "CPLE_None in CURL_INFO_HEADER_OUT: GET /sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif HTTP/1.1",
  "Host: sentinel-cogs.s3.us-west-2.amazonaws.com",
  "User-Agent: GDAL/3.7.3",
  "Accept: */*",
  "Range: bytes=11075584-12189695",
  "",
  "CPLE_None in CURL_INFO_HEADER_IN: HTTP/1.1 206 Partial Content",
  "CPLE_None in CURL_INFO_HEADER_IN: x-amz-id-2: dBrzELUWdBHGmu7gG4b5UgZ1SgZfl3nn4OF+bQjnPGRxHmUnjcHE24thEHncL6q6j2AixFF3GYI=",
  "CPLE_None in CURL_INFO_HEADER_IN: x-amz-request-id: 4B1HMCNGJGFWVKXB",
  "CPLE_None in CURL_INFO_HEADER_IN: Date: Thu, 18 Apr 2024 13:54:34 GMT",
  "CPLE_None in CURL_INFO_HEADER_IN: Last-Modified: Mon, 09 Oct 2023 01:23:09 GMT",
  "CPLE_None in CURL_INFO_HEADER_IN: ETag: \"3e23880cd7ac8a32ae71c55ad4f85c5d-24\"",
  "CPLE_None in CURL_INFO_HEADER_IN: x-amz-storage-class: INTELLIGENT_TIERING",
  "CPLE_None in CURL_INFO_HEADER_IN: x-amz-server-side-encryption: AES256",
  "CPLE_None in CURL_INFO_HEADER_IN: Cache-Control: public, max-age=31536000, immutable",
  "CPLE_None in CURL_INFO_HEADER_IN: Accept-Ranges: bytes",
  "CPLE_None in CURL_INFO_HEADER_IN: Content-Range: bytes 11075584-12189695/200959614",
  "CPLE_None in CURL_INFO_HEADER_IN: Content-Type: image/tiff; application=geotiff; profile=cloud-optimized",
  "CPLE_None in CURL_INFO_HEADER_IN: Server: AmazonS3",
  "CPLE_None in CURL_INFO_HEADER_IN: Content-Length: 1114112",
  "CPLE_None in CURL_INFO_HEADER_IN: ",
  "CPLE_None in CURL_INFO_TEXT: Connection #0 to host sentinel-cogs.s3.us-west-2.amazonaws.com left intact",
  "CPLE_None in VSICURL: Got response_code=206",
  "CPLE_None in GDAL: GDALWarpKernel()::GWKNearestByte() Src=1824,2039,351x130 Dst=0,128,348x128",
  "CPLE_None in GDAL: GDALWarpKernel()::GWKNearestByte() Src=1823,2167,352x94 Dst=0,256,348x92",
  "IO window xoff=0.0 yoff=0.0 width=1391.0 height=1391.0",
  "CPLE_None in GDAL: GDALClose(/vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif, this=0x11f249ca0)",
  "Exiting env context: <rasterio.env.Env object at 0x13fa59460>",
  "Cleared existing <rasterio._env.GDALEnv object at 0x13fa599d0> options",
  "Stopped GDALEnv <rasterio._env.GDALEnv object at 0x13fa599d0>.",
  "Exiting outermost env",
  "Exited env context: <rasterio.env.Env object at 0x13fa59460>"
]

@vincentsarago
Copy link
Member Author

Some weird stuff from the logs:

  • we have 2 HEAD (GetFileSize) requests to .xml and .XMLfiles
  • we have 1 GET (GetFileList) request

while we have GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR

CPLE_None in VSICURL: GetFileSize(https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif)=200959614  response_code=200
CPLE_None in VSICURL: GetFileList(/vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A)
CPLE_None in VSICURL: GetFileSize(https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.xml): response_code=404, server error msg=HTTP/1.1 404 Not Found
CPLE_None in VSICURL: GetFileSize(https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.XML): response_code=404, server error msg=HTTP/1.1 404 Not Found

@vincentsarago vincentsarago mentioned this issue Apr 18, 2024
Closed
@rouault
Copy link

rouault commented Apr 18, 2024

I don't confirm that using plain GDAL command line (3.8.5 or master):

$ CPL_VSIL_SHOW_NETWORK_STATS=YES GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR CPL_DEBUG=ON gdalinfo /vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif
HTTP: libcurl/7.68.0 GnuTLS/3.6.13 zlib/1.2.11 brotli/1.0.7 libidn2/2.2.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh/0.9.3/openssl/zlib nghttp2/1.40.0 librtmp/2.3
VSICURL: GetFileSize(https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif)=200959614  response_code=200
VSICURL: Downloading 0-16383 (https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif)...
VSICURL: Got response_code=206
GDAL: GDALOpen(/vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif, this=0x55daa42fd160) succeeds as GTiff.
GDAL: GDALDefaultOverviews::OverviewScan()
GTiff: ScanDirectories()
GTiff: Opened 5490x5490 overview.
GTiff: Opened 2745x2745 overview.
GTiff: Opened 1373x1373 overview.
GTiff: Opened 687x687 overview.
OGR_CT: Source CRS: 'EPSG:32615'
OGR_CT: Target CRS: 'GEOGCRS["WGS 84",DATUM["World Geodetic System 1984",ELLIPSOID["WGS 84",6378137,298.257223563,LENGTHUNIT["metre",1]],ID["EPSG",6326]],PRIMEM["Greenwich",0,ANGLEUNIT["degree",0.0174532925199433],ID["EPSG",8901]],CS[ellipsoidal,2],AXIS["geodetic latitude (Lat)",north,ORDER[1],ANGLEUNIT["degree",0.0174532925199433,ID["EPSG",9122]]],AXIS["geodetic longitude (Lon)",east,ORDER[2],ANGLEUNIT["degree",0.0174532925199433,ID["EPSG",9122]]]]'
Driver: GTiff/GeoTIFF
Files: /vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif
Size is 10980, 10980
Coordinate System is:
PROJCRS["WGS 84 / UTM zone 15N",
    BASEGEOGCRS["WGS 84",
        DATUM["World Geodetic System 1984",
            ELLIPSOID["WGS 84",6378137,298.257223563,
                LENGTHUNIT["metre",1]]],
        PRIMEM["Greenwich",0,
            ANGLEUNIT["degree",0.0174532925199433]],
        ID["EPSG",4326]],
    CONVERSION["UTM zone 15N",
        METHOD["Transverse Mercator",
            ID["EPSG",9807]],
        PARAMETER["Latitude of natural origin",0,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8801]],
        PARAMETER["Longitude of natural origin",-93,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8802]],
        PARAMETER["Scale factor at natural origin",0.9996,
            SCALEUNIT["unity",1],
            ID["EPSG",8805]],
        PARAMETER["False easting",500000,
            LENGTHUNIT["metre",1],
            ID["EPSG",8806]],
        PARAMETER["False northing",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8807]]],
    CS[Cartesian,2],
        AXIS["(E)",east,
            ORDER[1],
            LENGTHUNIT["metre",1]],
        AXIS["(N)",north,
            ORDER[2],
            LENGTHUNIT["metre",1]],
    USAGE[
        SCOPE["Navigation and medium accuracy spatial referencing."],
        AREA["Between 96°W and 90°W, northern hemisphere between equator and 84°N, onshore and offshore. Canada - Manitoba; Nunavut; Ontario. Ecuador -Galapagos. Guatemala. Mexico. United States (USA)."],
        BBOX[0,-96,84,-90]],
    ID["EPSG",32615]]
Data axis to CRS axis mapping: 1,2
Origin = (399960.000000000000000,5000040.000000000000000)
Pixel Size = (10.000000000000000,-10.000000000000000)
Metadata:
  OVR_RESAMPLING_ALG=AVERAGE
  AREA_OR_POINT=Area
Image Structure Metadata:
  COMPRESSION=DEFLATE
  INTERLEAVE=PIXEL
  PREDICTOR=2
Corner Coordinates:
Upper Left  (  399960.000, 5000040.000) ( 94d16'21.18"W, 45d 8'48.29"N)
Lower Left  (  399960.000, 4890240.000) ( 94d15' 4.02"W, 44d 9'30.66"N)
Upper Right (  509760.000, 5000040.000) ( 92d52'33.00"W, 45d 9'13.57"N)
Lower Right (  509760.000, 4890240.000) ( 92d52'40.53"W, 44d 9'55.08"N)
Center      (  454860.000, 4945140.000) ( 93d34' 9.68"W, 44d39'29.53"N)
Band 1 Block=1024x1024 Type=Byte, ColorInterp=Red
  NoData Value=0
  Overviews: 5490x5490, 2745x2745, 1373x1373, 687x687
Band 2 Block=1024x1024 Type=Byte, ColorInterp=Green
  NoData Value=0
  Overviews: 5490x5490, 2745x2745, 1373x1373, 687x687
Band 3 Block=1024x1024 Type=Byte, ColorInterp=Blue
  NoData Value=0
  Overviews: 5490x5490, 2745x2745, 1373x1373, 687x687
GDAL: GDALClose(/vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif, this=0x55daa42fd160)
GDAL: In GDALDestroy - unloading GDAL shared library.
Network statistics:
{
  "methods":{
    "HEAD":{
      "count":1
    },
    "GET":{
      "count":1,
      "downloaded_bytes":16384
    }
  },
  "handlers":{
    "vsicurl":{
      "methods":{
        "HEAD":{
          "count":1
        },
        "GET":{
          "count":1,
          "downloaded_bytes":16384
        }
      },
      "files":{
        "\/vsicurl\/https:\/\/sentinel-cogs.s3.us-west-2.amazonaws.com\/sentinel-s2-l2a-cogs\/15\/T\/VK\/2023\/10\/S2B_15TVK_20231008_0_L2A\/TCI.tif":{
          "methods":{
            "GET":{
              "count":1,
              "downloaded_bytes":16384
            }
          },
          "actions":{
            "Read":{
              "methods":{
                "GET":{
                  "count":1,
                  "downloaded_bytes":16384
                }
              }
            }
          }
        }
      },
      "actions":{
        "Stat":{
          "methods":{
            "HEAD":{
              "count":1
            }
          },
          "files":{
            "\/vsicurl\/https:\/\/sentinel-cogs.s3.us-west-2.amazonaws.com\/sentinel-s2-l2a-cogs\/15\/T\/VK\/2023\/10\/S2B_15TVK_20231008_0_L2A\/TCI.tif":{
              "methods":{
                "HEAD":{
                  "count":1
                }
              }
            }
          }
        }
      }
    }
  }
}

@vincentsarago
Copy link
Member Author

For some reason doing INFO works fine, it's when reading the data that we get additional HEAD/GET

info is tested here and works

rio-tiler/tests/benchmarks/requests.py

Lines 14 to 36 in 473de24

def test_info():
"""Info should only GET the header."""
@profile(
kernels=True,
config={
"GDAL_HTTP_MERGE_CONSECUTIVE_RANGES": "YES",
"GDAL_DISABLE_READDIR_ON_OPEN": "EMPTY_DIR",
"GDAL_INGESTED_BYTES_AT_OPEN": 32768,
},
quiet=True,
add_to_return=True,
)
def info(src_path: str):
with Reader(src_path) as src_dst:
return src_dst.info()
_, stats = info(dataset_url)
assert stats["HEAD"]["count"] == 1
assert stats["GET"]["count"] == 1
assert stats["GET"]["bytes"] == 32768
assert stats["GET"]["ranges"] == ["0-32767"]
assert not stats["WarpKernels"]

@vincentsarago
Copy link
Member Author

well gdalwarp gives the expected result

CPL_VSIL_SHOW_NETWORK_STATS=YES GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR CPL_DEBUG=ON gdalwarp -q /vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif out.tif -t_srs EPSG:3857 -ts 256 256 -te -10390543.876974 5518141.945963 -10370975.997733 5537709.825204 -overwrite
HTTP: libcurl/8.4.0 (SecureTransport) LibreSSL/3.3.6 zlib/1.2.12 nghttp2/1.58.0
VSICURL: GetFileSize(https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif)=200959614  response_code=200
VSICURL: Downloading 0-32767 (https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif)...
VSICURL: Got response_code=206
GDAL: GDALOpen(/vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif, this=0x10fe23d90) succeeds as GTiff.
GDAL: Using GTiff driver
GDAL: Computing area of interest: -94.2725, 44.1585, -92.8758, 45.1538
GDAL: GDALDriver::Create(GTiff,out.tif,256,256,3,Byte,0x10ffdced0)
WARP: Copying metadata from first source to destination dataset
GTiff: ScanDirectories()
GTiff: Opened 5490x5490 overview.
GTiff: Opened 2745x2745 overview.
GTiff: Opened 1373x1373 overview.
GTiff: Opened 687x687 overview.
WARP: Selecting overview level 1 for /vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif
GDAL: GDALOverviewDataset(/vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif, this=0x13100c690) creation.
WARP: srcNoData=0.000000 dstNoData=0.000000
WARP: calling GDALSetRasterNoDataValue() for band#0
WARP: srcNoData=0.000000 dstNoData=0.000000
WARP: calling GDALSetRasterNoDataValue() for band#1
WARP: srcNoData=0.000000 dstNoData=0.000000
WARP: calling GDALSetRasterNoDataValue() for band#2
GDALWARP: Defining SKIP_NOSOURCE=YES
GDAL: Computing area of interest: -94.2725, 44.1585, -92.8758, 45.1538
VSICURL: Downloading 9019424-10144695 (https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif)...
VSICURL: Downloading 11076968-12175156 (https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif)...
VSICURL: Download completed
GDAL: GDAL_CACHEMAX = 12288 MB
GDAL: GDALWarpKernel()::GWKNearestByte() Src=1823,1911,352x350 Dst=0,0,256x256
GTiff: Adjusted bytes to write from 7680 to 4608.
GDAL: GDALClose(out.tif, this=0x131005630)
GDAL: GDALClose(/vsicurl/https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/15/T/VK/2023/10/S2B_15TVK_20231008_0_L2A/TCI.tif, this=0x10fe23d90)
Network statistics:
{
  "methods":{
    "HEAD":{
      "count":1
    },
    "GET":{
      "count":2,
      "downloaded_bytes":2256229
    }
  },
  "handlers":{
    "vsicurl":{
      "methods":{
        "HEAD":{
          "count":1
        },
        "GET":{
          "count":2,
          "downloaded_bytes":2256229
        }
      },
      "files":{
        "\/vsicurl\/https:\/\/sentinel-cogs.s3.us-west-2.amazonaws.com\/sentinel-s2-l2a-cogs\/15\/T\/VK\/2023\/10\/S2B_15TVK_20231008_0_L2A\/TCI.tif":{
          "methods":{
            "GET":{
              "count":2,
              "downloaded_bytes":2256229
            }
          },
          "actions":{
            "Read":{
              "methods":{
                "GET":{
                  "count":1,
                  "downloaded_bytes":32768
                }
              }
            },
            "ReadMultiRange":{
              "methods":{
                "GET":{
                  "count":1,
                  "downloaded_bytes":2223461
                }
              }
            }
          }
        }
      },
      "actions":{
        "Stat":{
          "methods":{
            "HEAD":{
              "count":1
            }
          },
          "files":{
            "\/vsicurl\/https:\/\/sentinel-cogs.s3.us-west-2.amazonaws.com\/sentinel-s2-l2a-cogs\/15\/T\/VK\/2023\/10\/S2B_15TVK_20231008_0_L2A\/TCI.tif":{
              "methods":{
                "HEAD":{
                  "count":1
                }
              }
            }
          }
        }
      }
    }
  }
}

😬

@rouault
Copy link

rouault commented Apr 18, 2024

you may want to break into GTiffDataset::LookForProjectionFromXML() . Normally papszSiblingFiles = GetSiblingFiles() should be such that papszSiblingFiles != nullptr and papszSiblingFiles[0] == nullptr if GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR

@vincentsarago
Copy link
Member Author

just to summarize:

with gdalwarp we can 1 HEAD + 2 GET request, while with rio-tiler/rasterio we get 3 HEAD / 4 GET 😬

@vincentsarago
Copy link
Member Author

@rouault is LookForProjectionFromXML something that changed between gdal 3.6 and 3.7?

@rouault
Copy link

rouault commented Apr 19, 2024

is LookForProjectionFromXML something that changed between gdal 3.6 and 3.7?

it has been introduced in 3.7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rouault @vincentsarago