oisst cloud-ready

[mdsumner]

it's deflate compressed COGs, separate vars and so not a size improved from the netcdfs, but it's a reformat we don't have to do.

this includes the planetary computer signing, so the formatting is garbled:

gdalinfo "/vsicurl/https://noaacdr.blob.core.w                                                                                                                          indows.net/cogs/sea-surface-temperature-optimum-interpolation/202212/oisst-avhrr                                                                                                                          -v02r01.20221231-sst.tif?st=2023-05-21T09%3A19%3A55Z&se=2023-05-22T10%3A04%3A56Z                                                                                                                          &sp=rl&sv=2021-06-08&sr=c&skoid=c85c15d6-d1ae-42d4-af60-e2ca0f81359b&sktid=72f98                                                                                                                          8bf-86f1-41af-91ab-2d7cd011db47&skt=2023-05-22T09%3A19%3A54Z&ske=2023-05-29T09%3                                                                                                                          A19%3A54Z&sks=b&skv=2021-06-08&sig=7IDvxLdZa%2BBTOI2MGaLXaaxb16dIqbpOlaWo06p7ySI                                                                                                                          %3D"
Driver: GTiff/GeoTIFF
Files: /vsicurl/https://noaacdr.blob.core.windows.net/cogs/sea-surface-temperatu                                                                                                                          re-optimum-interpolation/202212/oisst-avhrr-v02r01.20221231-sst.tif?st=2023-05-2                                                                                                                          1T09%3A19%3A55Z&se=2023-05-22T10%3A04%3A56Z&sp=rl&sv=2021-06-08&sr=c&skoid=c85c1                                                                                                                          5d6-d1ae-42d4-af60-e2ca0f81359b&sktid=72f988bf-86f1-41af-91ab-2d7cd011db47&skt=2                                                                                                                          023-05-22T09%3A19%3A54Z&ske=2023-05-29T09%3A19%3A54Z&sks=b&skv=2021-06-08&sig=7I                                                                                                                          DvxLdZa%2BBTOI2MGaLXaaxb16dIqbpOlaWo06p7ySI%3D
Size is 1440, 720
Coordinate System is:
GEOGCRS["WGS 84",
    ENSEMBLE["World Geodetic System 1984 ensemble",
        MEMBER["World Geodetic System 1984 (Transit)"],
        MEMBER["World Geodetic System 1984 (G730)"],
        MEMBER["World Geodetic System 1984 (G873)"],
        MEMBER["World Geodetic System 1984 (G1150)"],
        MEMBER["World Geodetic System 1984 (G1674)"],
        MEMBER["World Geodetic System 1984 (G1762)"],
        MEMBER["World Geodetic System 1984 (G2139)"],
        ELLIPSOID["WGS 84",6378137,298.257223563,
            LENGTHUNIT["metre",1]],
        ENSEMBLEACCURACY[2.0]],
    PRIMEM["Greenwich",0,
        ANGLEUNIT["degree",0.0174532925199433]],
    CS[ellipsoidal,2],
        AXIS["geodetic latitude (Lat)",north,
            ORDER[1],
            ANGLEUNIT["degree",0.0174532925199433]],
        AXIS["geodetic longitude (Lon)",east,
            ORDER[2],
            ANGLEUNIT["degree",0.0174532925199433]],
    USAGE[
        SCOPE["Horizontal component of 3D system."],
        AREA["World."],
        BBOX[-90,-180,90,180]],
    ID["EPSG",4326]]
Data axis to CRS axis mapping: 2,1
Origin = (-180.000000000000000,90.000000000000000)
Pixel Size = (0.250000000000000,-0.250000000000000)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  COMPRESSION=DEFLATE
  INTERLEAVE=BAND
  LAYOUT=COG
Corner Coordinates:
Upper Left  (-180.0000000,  90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"N)
Lower Left  (-180.0000000, -90.0000000) (180d 0' 0.00"W, 90d 0' 0.00"S)
Upper Right ( 180.0000000,  90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"N)
Lower Right ( 180.0000000, -90.0000000) (180d 0' 0.00"E, 90d 0' 0.00"S)
Center      (   0.0000000,   0.0000000) (  0d 0' 0.01"E,  0d 0' 0.01"N)
Band 1 Block=512x512 Type=Int16, ColorInterp=Gray
  NoData Value=-999
  Overviews: 720x360, 360x180

So, get the last year of urls:

library(rstac)
s <- stac("https://planetarycomputer.microsoft.com/api/stac/v1")

datetime <-"2022-01-01/2022-12-31"
urls <- s %>% stac_search(collections = "noaa-cdr-sea-surface-temperature-optimum-interpolation", datetime = datetime) %>% 
  post_request() %>% assets_select(asset_names = c("sst")) %>% 
  items_sign(sign_fn = sign_planetary_computer()) %>% assets_url()

vsi <- file.path("/vsicurl", urls)
## this takes 462 seconds, I tried GDAL parallel threads but it's not working how I expect (or something)
system.time({
 mn <- gdal_raster_data(vsi, target_ext = c(-180, 180, -90, 90), target_res = 1, resample = "average")
})

pretty promising! This would be a drop-in replacement for our netcdfs for local use, but I think a size hit unless I've missed something (though we can skip the err, anom, ice probably)

[mdsumner]

not up to date though, and I can't get all the urls at once

noaacdr.blob.core.windows.net/cogs/sea-surface-temperature-optimum-interpolation/202305/oisst-avhrr-v02r01.20230506-sst.tif

range(sstfiles()$date)
[1] "1981-09-01 UTC" "2023-05-19 UTC"

can generate them all though, tested with a few hundred and furrr

(obviously shouldn't store this signing thing)

generate_sst_urls <- function(date) {
  year <- format(date, "%Y")
  month <- format(date, "%m")
  day <- format(date, "%d")
  out <- sprintf("https://noaacdr.blob.core.windows.net/cogs/sea-surface-temperature-optimum-interpolation/%s%s/oisst-avhrr-v02r01.%s%s%s-sst.tif",
          year, month, year, month, day)
  paste0(out, "?st=2023-05-21T12%3A24%3A23Z&se=2023-05-22T13%3A09%3A23Z&sp=rl&sv=2021-06-08&sr=c&skoid=c85c15d6-d1ae-42d4-af60-e2ca0f81359b&sktid=72f988bf-86f1-41af-91ab-2d7cd011db47&skt=2023-05-22T03%3A19%3A13Z&ske=2023-05-29T03%3A19%3A13Z&sks=b&skv=2021-06-08&sig=2Uw2DdDmG2MqnanRoAGjkKdmCe3zKaqLRLXQuoMxats%3D")
}

[mdsumner]

also check this out https://mastodon.nz/@weiji14/110217986392338955

https://zen3geo.readthedocs.io/en/v0.6.0/multi-resolution.html

[mdsumner]

sits is already navigating a lot of catalogues: https://github.com/e-sensing/sits