Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NASA FIRMS archive data does not integrate well with GDAL-based backend #264

Open
goergen95 opened this issue May 13, 2024 · 5 comments
Open

NASA FIRMS archive data does not integrate well with GDAL-based backend #264

goergen95 opened this issue May 13, 2024 · 5 comments
Assignees
@goergen95

Comments

@goergen95
Copy link
Member

goergen95 commented May 13, 2024 •
edited

While working on adding GDAL-based backend for resources and indicators (#240), the NASA FIRMS resource produces some troubles when it comes to automatic downloading.

With the new backend, we assume a resource function to return an sf object where each row represents the spatial extent of a GDAL-readable dataset. The resource is only fetched in case to local/cloud storage if specified, it is read directly from the source otherwise.

This behaviour does not integrate very well with how NASA FIRMS archive data is distributed. For each instrument, the data is bundled into yearly ZIP files that we can automatically fetch from here. Inside of those, there are country-specific CSV files with the data.

Theoretically, we could construct file paths such as:

"/vsizip//vsicurl/https://firms.modaps.eosdis.nasa.gov/data/country/zips/viirs-snpp_2019_all_countries.zip/viirs-snpp_2019_Afghanistan.csv"
"/vsizip//vsicurl/https://firms.modaps.eosdis.nasa.gov/data/country/zips/viirs-snpp_2019_all_countries.zip//viirs-snpp_2019_Albania.csv"
...

But we would need to construct footprints for each of those which would take a considerable amount of time while querying against a ZIP on a remote server. We could just model those extents, e.g. by querying a vector geometries of all countries of the world form another source, but it is really error prone if there are differences compared to how NASA FIRMS data is bundled.

I see two options:

    1. We need to find a data source which is distributed in a more suitable way to above mentioned requirements,
    1. we have to let users download the data manually from here, e.g. in GeoJSON
@goergen95 goergen95 changed the title NASA FIRMS archive data does integrate well with GDAL-based backend NASA FIRMS archive data does not integrate well with GDAL-based backend May 13, 2024
@goergen95 goergen95 self-assigned this May 13, 2024
@fBedecarrats
Copy link
Collaborator

fBedecarrats commented May 13, 2024 •
edited

I checked on https://github.com/opengeos/geospatial-data-catalogs, which references the major open geodata repositories. I see that a "FIRMS" product is accessible on GEE: https://github.com/opengeos/geospatial-data-catalogs
Another such entry is mentionned on the NASA CMR catalog here:

"MCD14DL_C5_NRT.v005 MODIS/Aqua+Terra Thermal Anomalies/Fire locations 1km FIRMS V005 NRT LM_FIRMS 2014-01-28 -180, -80, 180, 80 https://cmr.earthdata.nasa.gov/search/concepts/C1219768065-LM_FIRMS.json Near Real-Time (NRT) MODIS Thermal Anomalies / Fire locations processed by FIRMS (Fire Information for Resource Management System) - Land Atmosphere Near real time Capability for EOS (LANCE), using swath products (MOD14/MYD14) rather than the tiled MOD14A1 and MYD14A1 products. The thermal anomalies / active fire represent the center of a 1km pixel that is flagged by the MODIS MOD14/MYD14 Fire and Thermal Anomalies algorithm (Giglio 2003) as containing one or more fires within the pixel. This is the most basic fire product in which active fires and other thermal anomalies, such as volcanoes, are identified.MCD14DL are available in the following formats: TXT, SHP, KML, WMS. These data are also provided through the FIRMS Fire Email Alerts. Please note only the TXT and SHP files contain all the attributes. not-provided"

I find no FIRMS entry on the other catalogs (AWS, Stac or planetary computer)

@goergen95
Copy link
Member Author

Thanks! I'll check NASA CMR since if I recall correctly, the GEE catalogs require a user account wit Google.

@goergen95
Copy link
Member Author

There are also some fire-related MODIS raster product on MPC:

@goergen95
Copy link
Member Author

Accessing the monthly COGs looks promising:

> gdalinfo "/vsicurl?pc_url_signing=yes&url=https://modiseuwest.blob.core.windows.net/modis-061-cogs/MCD64A1/35/08/2024032/MCD64A1.A2024032.h35v08.061.2024100033846_Burn_Date.tif"

Driver: GTiff/GeoTIFF
Files: /vsicurl?pc_url_signing=yes&url=https://modiseuwest.blob.core.windows.net/modis-061-cogs/MCD64A1/35/08/2024032/MCD64A1.A2024032.h35v08.061.2024100033846_Burn_Date.tif
Size is 2400, 2400
Coordinate System is:
PROJCRS["unnamed",
    BASEGEOGCRS["Unknown datum based upon the custom spheroid",
        DATUM["Not specified (based on custom spheroid)",
            ELLIPSOID["Custom spheroid",6371007.181,0,
                LENGTHUNIT["metre",1,
                    ID["EPSG",9001]]]],
        PRIMEM["Greenwich",0,
            ANGLEUNIT["degree",0.0174532925199433,
                ID["EPSG",9122]]]],
    CONVERSION["Sinusoidal",
        METHOD["Sinusoidal"],
        PARAMETER["Longitude of natural origin",0,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8802]],
        PARAMETER["False easting",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8806]],
        PARAMETER["False northing",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8807]]],
    CS[Cartesian,2],
        AXIS["easting",east,
            ORDER[1],
            LENGTHUNIT["metre",1,
                ID["EPSG",9001]]],
        AXIS["northing",north,
            ORDER[2],
            LENGTHUNIT["metre",1,
                ID["EPSG",9001]]]]
Data axis to CRS axis mapping: 1,2
Origin = (18903158.834352001547813,1111950.519662000006065)
Pixel Size = (463.312716527916507,-463.312716527916677)
Metadata:
  ALGORITHMPACKAGEACCEPTANCEDATE=2016-03-01
  ALGORITHMPACKAGEMATURITYCODE=Normal
  ALGORITHMPACKAGENAME=MOD_PR64A1
  ALGORITHMPACKAGEVERSION=6
  AQUADATAUSED=Yes
  AREA_OR_POINT=Area
  ASSOCIATEDINSTRUMENTSHORTNAME.1=MODIS
  ASSOCIATEDINSTRUMENTSHORTNAME.2=MODIS
  ASSOCIATEDPLATFORMSHORTNAME.1=Terra
  ASSOCIATEDPLATFORMSHORTNAME.2=Aqua
  ASSOCIATEDSENSORSHORTNAME.1=MODIS
  ASSOCIATEDSENSORSHORTNAME.2=MODIS
  AUTOMATICQUALITYFLAG.1=Passed
  AUTOMATICQUALITYFLAGEXPLANATION.1=No automatic quality assessment is performed in the PGE
  BurnedCells=0
  CHARACTERISTICBINANGULARSIZE=15.0
  CHARACTERISTICBINSIZE=463.312716527778
  CodeVersion=4.0.1
  DATACOLUMNS=2400
  DATAROWS=2400
  DAYNIGHTFLAG=Day
  DESCRREVISION=6.1
  EASTBOUNDINGCOORDINATE=179.9999999838
  EXCLUSIONGRINGFLAG.1=N
  GEOANYABNORMAL=false
  GEOESTMAXRMSERROR=50.0
  GLOBALGRIDCOLUMNS=86400
  GLOBALGRIDROWS=43200
  GRINGPOINTLATITUDE.1=-0.0, 9.9911293251, 9.9989779081, -0.00688902
  GRINGPOINTLONGITUDE.1=169.9999999847, 169.9285928411, -179.9284766135, 179.9998921159
  GRINGPOINTSEQUENCENO.1=1, 2, 3, 4
  HDFEOSVersion=HDFEOS_V2.19
  HORIZONTALTILENUMBER=35
  identifier_product_doi=10.5067/MODIS/MCD64A1.061
  identifier_product_doi_authority=https://doi.org
  INPUTPOINTER=/MODAPSops8/archive/f20227/running/AMPM_C61_L26glu/264310649/MCD64A0.A2024032.h35v08.061.2024100033836.hdf, /MODAPSops8/PGE/AMPM/coeff/PGE134/MOD_PR64A1/MCD12Q1.2013.051-UMD-clean-4km.hdf
  LandCells=1677
  LC_input_file=MCD12Q1.2013.051-UMD-clean-4km.hdf
  LOCALGRANULEID=MCD64A1.A2024032.h35v08.061.2024100033846.hdf
  LOCALVERSIONID=4.0.1
  LONGNAME=MODIS/Terra+Aqua Direct Broadcast Burned Area Monthly L3 Global 500m SIN Grid
  long_name=ordinal day of burn
  MCD64A0_input_file=MCD64A0.A2024032.h35v08.061.2024100033836.hdf
  MissingCells=524362
  NADIRDATARESOLUTION=500m
  NORTHBOUNDINGCOORDINATE=9.9999999991
  NUMBERBURNEDPIXELS=0
  PARAMETERNAME.1=MCD64A1
  PGEVERSION=6.1.4
  PROCESSINGCENTER=MODAPS
  PROCESSINGENVIRONMENT=Linux minion20227 5.4.0-1093-fips #103-Ubuntu SMP Thu Feb 8 14:02:37 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
  ProductEndDay=60
  PRODUCTIONDATETIME=2024-04-09T07:38:46.000Z
  ProductStartDay=32
  QAPERCENTCLOUDCOVER.1=0
  QAPERCENTINTERPOLATEDDATA.1=0
  QAPERCENTMISSINGDATA.1=61
  QAPERCENTOUTOFBOUNDSDATA.1=0
  RANGEBEGINNINGDATE=2024-02-01
  RANGEBEGINNINGTIME=00:00:00
  RANGEENDINGDATE=2024-02-29
  RANGEENDINGTIME=23:59:59
  REPROCESSINGACTUAL=reprocessed
  REPROCESSINGPLANNED=further update is anticipated
  SCIENCEQUALITYFLAG.1=Not Investigated
  SCIENCEQUALITYFLAGEXPLANATION.1=See http://landweb.nascom.nasa.gov/cgi-bin/QA_WWW/qaFlagPage.cgi?sat=aqua&ver=C6 for the product Science Quality status.
  SHORTNAME=MCD64A1
  SOUTHBOUNDINGCOORDINATE=-0.0
  SPSOPARAMETERS=2015
  TERRADATAUSED=Yes
  tile=h35v08
  TileID=51035008
  ValidLandCells=655
  valid_range=0, 366
  VERSIONID=61
  VERTICALTILENUMBER=08
  water=-2
  WESTBOUNDINGCOORDINATE=169.9999999847
  year=2024
  _FillValue=-1
Image Structure Metadata:
  COMPRESSION=DEFLATE
  INTERLEAVE=BAND
  LAYOUT=COG
Corner Coordinates:
Upper Left  (18903158.834, 1111950.520) (172d37'21.09"E, 10d 0' 0.00"N)
Lower Left  (18903158.834,      -0.000) (170d 0' 0.00"E,  0d 0' 0.00"S)
Upper Right (20015109.354, 1111950.520) (177d13'23.56"W, 10d 0' 0.00"N)
Lower Right (20015109.354,      -0.000) (180d 0' 0.00"E,  0d 0' 0.00"S)
Center      (19459134.094,  555975.260) (175d40' 6.50"E,  5d 0' 0.00"N)
Band 1 Block=512x512 Type=Int16, ColorInterp=Gray
  Description = ordinal day of burn
  NoData Value=-1
  Overviews: 1200x1200, 600x600, 300x300

I'll have a dive into the product documentation and will evaluate if we can use that product to derive and indicator of e.g. monthly burned area. This will be different from the original one, but hopefully (more?) useful.

@goergen95
Copy link
Member Author

@Jo-Schie and @karpfen: a36a8f7 adds an initial draft of a burned area indicator based on the MCD64A1 on the dev branch in case you want to test it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@goergen95 goergen95

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@fBedecarrats @goergen95