GeoTIFF with invalidated optimizations shows up as valid

[mplough-kobold]

Steps to reproduce with rio-cogeo==3.5.1:

Use the files in input and output data.zip or get/generate them as follows.

Obtain lcc-datum.tif from https://download.osgeo.org/geotiff/samples/made_up/.

Generate a slightly invalid COG lcc-datum-cog.tif as follows. Note that while COG does not support creation option TILED, this does run to completion generate a GeoTIFF.

Package versions:
dask==2023.4.0
distributed==2023.4.0
rioxarray==0.14.0
xarray==2023.3.0

import rioxarray
import xarray as xr
from dask.distributed import Client, LocalCluster, Lock

cluster = LocalCluster(n_workers=4)
client = Client(cluster)

chunks = {"x": 128, "y": 128}
datum = xr.open_dataarray("lcc-datum.tif", chunks=chunks)
datum.rio.to_raster("lcc-datum-cog.tif", driver="COG", tiled=True, lock=Lock("rio", client=client))

client.close()
cluster.close()

Run rio cogeo validate lcc-datum-cog.tif and view the output:

$ rio cogeo validate lcc-datum-cog.tif
WARNING:rasterio._env:CPLE_AppDefined in lcc-datum-cog.tif: This file used to have optimizations in its layout, but those have been, at least partly, invalidated by later changes
WARNING:rasterio._env:CPLE_AppDefined in /absolute/path/to/lcc-datum-cog.tif: This file used to have optimizations in its layout, but those have been, at least partly, invalidated by later changes
/absolute/path/to/lcc-datum-cog.tif is a valid cloud optimized GeoTIFF

Is this file a valid COG? Based on the warnings it looks like there are issues in the file's layout that rio-cogeo does not detect.

[vincentsarago]

@mplough-kobold COG is a CreateCopy only driver (https://gdal.org/drivers/raster/cog.html#driver-capabilities), when you are using rio.to_raster it will create a COG in Create mode which is theory is not the best way to create COG but rasterio allows it.

If the validate step works I believe it's fine but to be 💯 I wouldn't use the to_raster method to create COG. Sadly rioxarray do not expose the copy method from rasterio

[mplough-kobold]

I definitely agree that the to_raster method is not a great way to create a COG; I was mainly using it here as a convenient way to generate data for a minimum reproducible example.

I've finally upgraded my TIFF skills and have a bit more knowledge of the file format.

Looking at the two files here, GDAL adds KNOWN_INCOMPATIBLE_EDITION=YES in the header ghost area of the output file lcc-datum-cog.tif -- that's where the warnings are coming from. GDAL's validate_cloud_optimized_geotiff.py keys off that header. Should rio-cogeo do the same?

Interestingly, running this code (same as above, just with compress="DEFLATE" added) produces a file that's 1.4 MB in size, while running rio cogeo create lcc-datum.tif rio-cogeo-create.tif produces output that's about 156 KB.

import rioxarray
import xarray as xr
from dask.distributed import Client, LocalCluster, Lock

cluster = LocalCluster(n_workers=4)
client = Client(cluster)

chunks = {"x": 128, "y": 128}
datum = xr.open_dataarray("lcc-datum.tif", chunks=chunks)
datum.rio.to_raster("lcc-datum-cog-deflate.tif", driver="COG", compress="DEFLATE", tiled=True, lock=Lock("rio", client=client))

client.close()
cluster.close()

Both files have TileByteCounts set to 156060 (decimal). However, lcc-datum-cog-deflate.tif has TileOffsets set to 1201560 (decimal) while rio-cogeo-create.tif has TileOffsets set to 933 (decimal). There's 1.2 megs of inaccessible "garbage" between the start of the image and the start of the image data.

It will likely be difficult to detect or validate the presence of that garbage data -- what if there's some tag that we don't know about that points to it? But GDAL knows that something bad happened here and it would be easy to parse the GDAL's ghost header if it's present.

Would you be amenable to either examining or accepting a PR that examines ghost headers, if present, and marks the file as invalid if the KNOWN_INCOMPATIBLE_EDITION is set to YES?