STAC Catalog: best practice to catalog contextual data?

[rogerkuou]

One notable feature of stmtools is its ability to enrich STM (Space-Time Matrix) with contextual data. This implies that stmtools should interface with various contextual datasets in the background. If these contextual datasets can be cataloged using a standardized metadata format, stmtools can initially query the catalog, then decide if needed to dive into the dataset. For instance, from the catalog, it can determine whether the contextual data contains information relevant to the specific-area, time-frame, or attributes of interest. This approach can be more efficient compared to directly interacting with each contextual dataset.

In this discussion, we aim to explore an example of such catalog technology: SpatioTemporal Asset Catalog (STAC), and assess its suitability for this purpose.

We assume that the user has established a local data repository containing a variety of contextual datasets.

[rogerkuou]

Initial issue for investigation created: #64, where we start with two specific datasets:

• BAG (Building registrations. Represents large volume polygons, 3GB in zip file, 7GB in gpkg)
• KNMI weather station datasets. Quoting Freek:

This is the link:

https://www.daggegevens.knmi.nl/klimatologie/uurgegevens

The idea would be to insert meteo data into the STM of the data (e.g. temperature, pressure) closest to the SAR satellite acquisition. (But maybe also: the sum of rainfall in the last 3 days, etc).

Furthermore, I attached:

an example data file (just 1000 lines) knmi_stn280_example_file.txt
two old Matlab scripts, handling the data. importMeteodata_knmi.zip
, get_meteo.zip
.csv file with KNMI station locations knmistations.csv (see also https://github.com/MotionbyLearning/sandbox/tree/develop_vector_processing for an example implementation using these station locations made earlier (in the 'step 3' script).

[rogerkuou]

Initial example created in PR: MotionbyLearning/example_scripts#3.
Three example notebooks have been created:

• Convert data to standard formats
• Build catalog
• Query catalog

Things to follow up:

1. Work with Parquet, quote Francesco:

Nice! One comment on the conversion notebook (we have already discussed it this morning). For handling parquet/geoparquet file, you could check out some of the blog posts from Chris Holmes (see here), who has converted some large Google building dataset to geoparquet (see repository, especially the "processing" section of the README). He is advocating a lot for DuckDB, maybe it could be worth to try it out.
Also a new geoparquet release is coming up: https://medium.com/radiant-earth-insights/geoparquet-1-1-coming-soon-9b72c900fbf2

2. Handle CRS system

To specify the asset projection, there is a dedicated STAC extension: https://github.com/stac-extensions/projection . Also, I think that is a STAC norm to use WGS84 for the geometry/bbox of the items, whatever the projection of the linked assets. This also allows one to search a catalog that has items in different projections. Also note that the projection metadata is missing for the KNMI dataset.

3. Make catalog pystac_client querible:

Unfortunately, I think you cannot use the search functionality of pystac_client on a static catalog (it only works for APIs).
So one needs to setup a server, example implementations are stac-server and stac-fastapi. Also this discussion on the topic of catalogs that needs to be often updated might be interesting. Just for visualization of a catalog via the web browser, one can use the STAC browser, which can be pointed to a static catalog. However, this does not seem to work with the public view of Spider (it works for instance with a static catalog on GitHub - try paste this link into the STAC browser).