[WIP] t.stac.import: Add STAC API import functionality #802

cwhite911 · 2022-09-30T18:53:17Z

I'm implementing STAC API import functionality into GRASS using pystac-client.

Currently, I'm modeling the module off of the STAC API request parameters:

collections – List of one or more Collection IDs
ids – List of one or more Item ids to filter on.
limit – A recommendation to the service as to the number of items to return per page of results. Defaults to 100.
max_items – The maximum number of items to return from the search, even if there are more matching results.
bbox – A list, tuple, or iterator representing a bounding box of 2D or 3D coordinates. Results will be filtered to only those intersecting the bounding box.
intersects – A string or dictionary representing a GeoJSON geometry, or an object that implements a geo_interface property, as supported by several libraries including Shapely, ArcPy, PySAL, and geojson. Results filtered to only those intersecting the geometry.
datatime – Either a single datetime or datetime range used to filter results.
query – List or JSON of query parameters as per the STAC API query extension
filter – JSON of query parameters as per the STAC API filter extension
filter_lang – Language variant used in the filter body. If filter is a dictionary or not provided, defaults to ‘cql2-json’. If filter is a string, defaults to cql2-text.

Details docs at https://pystac-client.readthedocs.io

I'm thinking we should use the current computational region as the default bbox to filter the request and provide an option to define a raster or vector to use as the intersects parameter.

For outputs, we have a few options.

Import all items individually.
Import a single patched raster.
Import as a STRDS.

Another option, instead of having a single module r.in.stac, we could split it into multiple modules and include specific STAC extension parameters in each:

r.in.stac
i.in.stac
v.in.stac

Thoughts...

metzm · 2022-10-04T07:28:18Z

When importing parts of a space-time collection, the corresponding output would be a GRASS space-time dataset, created with a t.* module, e.g. t.in.stac.

veroandreo · 2022-10-04T11:15:21Z

When importing parts of a space-time collection, the corresponding output would be a GRASS space-time dataset, created with a t.* module, e.g. t.in.stac.

What about simplifying all into:

t.rast.in.stac or t.in.stac.rast
t.vect.in.stac or t.in.stac.vect

or a toolset t.in.stac with raster and vector options as submodules?

STAC stands for spatio-temporal after all, and that's the beauty of it, no? In any case, big 👍 for this!!

ninsbl

Nice. Good to see you are working on STAC.

I have been working on a module to import space-time data in NetCDF format here:
#794

Maybe you can get some inspiration and pick useful code snippets. If so, we may consider moving some of them into a library...

ninsbl · 2022-10-02T14:42:36Z

src/raster/r.in.stac/r.in.stac.py

+# % guisection: Request
+# %end
+
+# %option


Isn` t that the same as the "bbox" option?

ninsbl · 2022-10-02T14:48:41Z

src/raster/r.in.stac/r.in.stac.py

+# %option G_OPT_V_INPUT
+# % key: intersects
+# % description: Results filtered to only those intersecting the geometry.
+# % guisection: Request


Are there any other predefined options? If so, it would be good to provide them in an options list, if "intersects" it the only valid option, implementing this as a flag for intersection might be a simpler choice

ninsbl · 2022-10-05T08:40:42Z

When importing parts of a space-time collection, the corresponding output would be a GRASS space-time dataset, created with a t.* module, e.g. t.in.stac.

What about simplifying all into:

t.rast.in.stac or t.in.stac.rast

t.vect.in.stac or t.in.stac.vect

or a toolset t.in.stac with raster and vector options as submodules?

STAC stands for spatio-temporal after all, and that's the beauty of it, no? In any case, big 👍 for this!!

Agreed. As for module naming, there is t.rast.import and t.vect.import in core, so it might be more consistent with existing approaches to name modules t.rast.import.stac and t.vect.import.stac (though three dots are not standard either)...

P.S.: Sorry @veroandreo for accidentally editing your comment...

cwhite911 · 2022-11-12T01:36:33Z

Thanks @veroandreo and @ninsbl for the feedback. We could also call the module t.stac.import with a raster or vector option that uses the sub-modules t.in.stac.rast or t.in.stac.vec in the background.

Another thing to consider is the t.out.stac. I've started a proposal for a GRASS STAC extension using the nc_spm_08 sample dataset. My thought is that I would like to be able to describe a grassdata directory as a STAC Catalog that contains Location STAC Collections that contain mapset STAC collections that have assets. This would allow sharing GRASS datasets as explorable metadata that can be viewed in any STAC Viewer or OpenPlains.

GRASS STAC Extenion: https://github.com/tomorrownow/grass-stac-extension

Here are some initial tests, but I'm still working on the first spec and want to get wider community feedback once the initial use cases are flushed out.

grassdata: STAC Catalog

{
    "type": "Catalog",
    "id": "grassdata",
    "stac_version": "1.0.0",
    "description": "GRASS GIS STAC data catalog used by OpenPlains.",
    "links": [
      {
        "rel": "root",
        "href": "https://example.com/grass_catalog/catalog.json",
        "type": "application/json"
      },
      {
        "rel": "child",
        "href": "https://example.com/grass_catalog/nc_spm_08/collection.json",
        "type": "application/json",
        "title": "nc_spm_08"
      },
      {
        "rel": "self",
        "href": "https://example.com/grass_catalog/catalog.json",
        "type": "application/json"
      }
    ],
    "stac_extensions": []
  }

nc_spm_08 location: STAC Collection

{
    "type": "Collection",
    "id": "nc_spm_08",
    "stac_version": "1.0.0",
    "description": "GRASS GIS Sample Datasets",
    "links": [
      {
        "rel": "root",
        "href": "https://example.com/grass_catalog/catalog.json",
        "type": "application/json"
      },
      {
        "rel": "child",
        "href": "https://example.com/grass_catalog/nc_spm_08/PERMANENT/collection.json",
        "type": "application/json",
        "title": "PERMANENT"
      },
      {
        "rel": "parent",
        "href": "https://example.com/grass_catalog/catalog.json",
        "type": "application/json"
      }
    ],
    "stac_extensions": [
      "https://stac-extensions.github.io/projection/v1.0.0/schema.json",
      "https://stac-extensions.github.io/scientific/v1.0.0/schema.json"
    ],
    "grass:type": "location",
    "proj:epsg": 3358,
    "sci:citation": "GRASS Development Team, 2022. Geographic Resources Analysis Support System (GRASS) Software, Version 8.0. Open Source Geospatial Foundation. https://grass.osgeo.org",
    "title": "nc_spm_08",
    "extent": {
      "bbox": [
        [
          33.83,
          -84.33,
          36.59,
          -75.38
        ]
      ]
    },
    "license": "GNU General Public License (GPL)",
    "keywords": [
      "GRASS GIS",
      "Location"
    ]
  }

PERMANENT mapset: STAC Collection

{
    "type": "Collection",
    "id": "PERMANENT",
    "stac_version": "1.0.0",
    "description": "defualt mapset",
    "links": [
      {
        "rel": "root",
        "href": "https://example.com/grass_catalog/catalog.json",
        "type": "application/json"
      },
      {
        "rel": "item",
        "href": "https://example.com/grass_catalog/nc_spm_08/PERMANENT/elevation/elevation.json",
        "type": "application/json"
      },
      {
        "rel": "parent",
        "href": "https://example.com/grass_catalog/nc_spm_08/collection.json",
        "type": "application/json",
        "title": "nc_spm_08"
      }
    ],
    "stac_extensions": [
      "https://stac-extensions.github.io/projection/v1.0.0/schema.json",
      "https://stac-extensions.github.io/scientific/v1.0.0/schema.json"
    ],
    "grass:type": "mapset",
    "proj:epsg": 3358,
    "sci:citation": "GRASS Development Team, 2022. Geographic Resources Analysis Support System (GRASS) Software, Version 8.0. Open Source Geospatial Foundation. https://grass.osgeo.org",
    "title": "PERMANENT",
    "extent": {
      "bbox": [
        [
          33.83,
          -84.33,
          36.59,
          -75.38
        ]
      ]
    },
    "license": "GNU General Public License (GPL)",
    "keywords": [
      "GRASS GIS",
      "mapset",
      "PERMANENT"
    ]
 }

Raster data: STAC Item

{
    "type": "Feature",
    "stac_version": "1.0.0",
    "id": "elevation",
    "properties": {
      "title": "\"South-West Wake county: Elevation NED 10m\"",
      "description": "\"generated by r.proj\"",
      "proj:epsg": 3358,
      "grass:datatype": "FCELL",
      "grass:comments": "\"r.proj input=\"ned03arcsec\" location=\"northcarolina_latlong\" mapset=\"\\helena\" output=\"elev_ned10m\" method=\"cubic\" resolution=10\"",
      "grass:creator": "\"helena\"",
      "grass:ewres": "10",
      "grass:nsres": "10",
      "grass:cols": "1500",
      "grass:location": "nc_spm_08",
      "grass:mapset": "PERMANENT",
      "grass:map": "elevation",
      "grass:maptype": "raster",
      "grass:min": "55.57879",
      "grass:max": "156.3299",
      "grass:ncats": "255",
      "grass:semantic_label": "\"none\"",
      "grass:source1": "\"\"",
      "grass:source2": "\"\"",
      "datetime": "2006-07-11T01:09:51Z"
    },
    "geometry": {
      "type": "Polygon",
      "coordinates": [
        [
          [
            645000.0,
            215000.0
          ],
          [
            645000.0,
            228500.0
          ],
          [
            630000.0,
            228500.0
          ],
          [
            630000.0,
            215000.0
          ],
          [
            645000.0,
            215000.0
          ]
        ]
      ]
    },
    "links": [
      {
        "rel": "root",
        "href": "https://example.com/grass_catalog/catalog.json",
        "type": "application/json"
      },
      {
        "rel": "collection",
        "href": "https://example.com/grass_catalog/nc_spm_08/PERMANENT/collection.json",
        "type": "application/json",
        "title": "PERMANENT"
      },
      {
        "rel": "parent",
        "href": "https://example.com/grass_catalog/nc_spm_08/PERMANENT/collection.json",
        "type": "application/json",
        "title": "PERMANENT"
      }
    ],
    "assets": {
      "raster": {
        "href": "/api/v3/locations/nc_spm_08/mapsets/PERMANENT/raster_layers/elevation",
        "type": "image/tiff; application=geotiff; profile=cloud-optimized",
        "title": "\"South-West Wake county: Elevation NED 10m\"",
        "roles": [
          "data"
        ]
      },
      "thumbnail": {
        "href": "/api/v3/locations/nc_spm_08/mapsets/PERMANENT/raster_layers/elevationrender",
        "type": "image/png",
        "title": "\"South-West Wake county: Elevation NED 10m\" Thumbnail",
        "roles": [
          "thumbnail"
        ]
      }
    },
    "bbox": [
        630000,
        215000,
        645000,
        228500
    ],
    "stac_extensions": [],
    "collection": "PERMANENT"
}

lucadelu · 2023-04-27T12:56:22Z

@cwhite911 do you have any updates on this?

This module could be really useful

veroandreo · 2023-06-05T10:49:59Z

@cwhite911 any plans to continue with this soon(ish)? Now that ESA will change data delivery, having access via STAC would be really relevant for many

wenzeslaus · 2023-09-19T01:08:07Z

Please preserve your shell and Git history. I want to see what happened to change the instructions accordingly.

neteler · 2023-09-19T08:03:35Z

src/temporal/t.stac.import/t.stac.import.py

+    avaliable_collections_ids = [c.id for c in list(avaliable_collections)]
+
+    gs.warning(_(f"Avaliable Collections: {avaliable_collections_ids}"))
+
+    if collections in avaliable_collections_ids:


Typo in various lines:

avaliable -> available

Avaliable -> Available

veroandreo · 2023-09-19T11:50:03Z

src/temporal/t.stac.import/README.md

+
+**t.stac.import** utilizes the
+[pystac-client (v0.5.1)](https://github.com/stac-utils/pystac-client) to search
+STAC APIs and import and import items into GRASS GIS.


remove one of the two "and import"

ninsbl · 2023-09-19T13:01:03Z

src/temporal/t.stac.import/t.stac.import.py

-    if collections in avaliable_collections:
+    avaliable_collections_ids = [c.id for c in list(avaliable_collections)]
+
+    gs.warning(_(f"Avaliable Collections: {avaliable_collections_ids}"))


I think translatable messages should not be f-strings, but rather something like this:

Suggested change

gs.warning(_(f"Avaliable Collections: {avaliable_collections_ids}"))

gs.warning(_("Avaliable Collections: {}").format(avaliable_collections_ids))

ninsbl · 2023-09-19T13:02:47Z

Really good to see this moving forward, especially with: https://dataspace.copernicus.eu/ providing Sentinel data in STAC format...

neteler · 2023-09-19T15:38:18Z

This PR needs a big rebase...

cwhite911 · 2023-09-19T17:55:11Z

This PR needs a big rebase...

Fixed the issue

wenzeslaus · 2023-09-19T18:43:29Z

Now, I finally understood (thanks @cwhite911!) how this thing with many commits happens. You have outdated branch. You decide to update it to the base branch (here grass8) by rebase. You do that. Then you push. You get a message that the remote branch and local branch diverged and that the push is not possible. The suggestion is to update you local branch. You decide to update the local branch from the remote one by rebase. And that's where the mess happens. All the perfectly fine commits from grass8 branch which are now on your local branch get removed and then re-applied on top of the latest commit on the remote branch. Then you happily push and then see all these extra commits duplicating changes already on the base branch.

The right operation after rebasing the local branch to the base branch is to force push. You changed all the commits on the local branch and that's what you want on the remote one too. Force push is the right operation here because you want to replace remote branch with what you have locally.

Git does not know that's what you are doing. It sees different commit hashes and it gives you advice which would preserve all these commits.

It is worth noting that, unlike merge, rebase changes the commit hashes. So, even the same change after a rebase, has a different hash, so it looks like a different commit to Git.

In light of Git giving the "wrong advice" in the rebase workflow, it might a good idea to use merge in all contributor workflows. I mean to have it in the instructions. There is no reason not to use rebase in general. One issue with this is terminology, merge can by misleading as it is also used in "merge a PR".

Sorry, for an off-topic post here, I'll follow up on this in a new issue or PR.

src/temporal/t.stac.import/t.stac.import.py

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

cwhite911 changed the title ~~[WIP] r.in.stac: Add STAC API import funtionality to GRASS~~ [WIP] r.in.stac: Add STAC API import funtionality Sep 30, 2022

cwhite911 changed the title ~~[WIP] r.in.stac: Add STAC API import funtionality~~ [WIP] r.in.stac: Add STAC API import functionality Sep 30, 2022

ninsbl reviewed Oct 5, 2022

View reviewed changes

ninsbl mentioned this pull request Mar 30, 2023

[Bug] i.sentinel.download: Support Copernicus Data Space Ecosystem Service #883

Open

cwhite911 changed the title ~~[WIP] r.in.stac: Add STAC API import functionality~~ [WIP] t.stac.import: Add STAC API import functionality Sep 19, 2023

neteler reviewed Sep 19, 2023

View reviewed changes

veroandreo reviewed Sep 19, 2023

View reviewed changes

ninsbl reviewed Sep 19, 2023

View reviewed changes

cwhite911 force-pushed the r.in.stac branch from 8b51fb6 to 18326d5 Compare September 19, 2023 15:23

cwhite911 and others added 8 commits September 19, 2023 13:48

r.in.stac: inital setup of module structure options and flags

e5a9b7e

Fixed formatting issues

fe59675

r.in.stac: added queries for collections and items

df8a3b3

Organized options

7ca8170

t.stac.import: Changed module name to t.stac.import

1ad5487

t.stac.import: Changed module name to t.stac.import

7cb2222

renamed grass to gs

c3192ce

added get items function

98b82b9

cwhite911 force-pushed the r.in.stac branch from 18326d5 to 98b82b9 Compare September 19, 2023 17:53

Added filter by bbox and user region

3172c51

Corey White and others added 16 commits September 19, 2023 16:25

Started the import and download fucntionallity

83feb9c

fixed typo

167412d

Added download method for COGs via https, s3, and gs

d39c6ce

Added query docs

c410e13

Updated multiprocessing options

e16cf83

Started added static catalog functionality

f4c31e9

Updated docs

c16bef9

Updated documentation

1072943

Added authentication methods

5af75ed

Added planetary computer support

911afe5

Cleaned code formatting

f75886c

Added ability to filter items assets by asset ids

675fbe7

Improved collection outputs

588eb08

Added additional file types

bf27b55

Started restructuring code into different modules

1d8ef0f

Merge branch 'grass8' into r.in.stac

812c473

github-actions bot reviewed Mar 29, 2024

View reviewed changes

cwhite911 and others added 10 commits March 28, 2024 22:57

Update src/temporal/t.stac.import/t.stac.import.py

f94b5a1

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Update src/temporal/t.stac.import/t.stac.import.py

5fb176a

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Update src/temporal/t.stac.import/t.stac.import.py

33e9b78

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Update src/temporal/t.stac.import/t.stac.import.py

9fdcc50

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Merge remote-tracking branch 'origin/stac.import' into r.in.stac

f33217c

Added t.stac.item

bc72965

Updated dependencies

a14d923

Fixed tests

97ea4e1

Merge branch 'grass8' into r.in.stac

802fb27

Merge branch 'grass8' into r.in.stac

8636b0d

wenzeslaus mentioned this pull request May 21, 2024

d.redraw: Rewritten in C OSGeo/grass#3484

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] t.stac.import: Add STAC API import functionality #802

[WIP] t.stac.import: Add STAC API import functionality #802

cwhite911 commented Sep 30, 2022

metzm commented Oct 4, 2022

veroandreo commented Oct 4, 2022 •

edited by ninsbl

ninsbl left a comment

ninsbl Oct 2, 2022

ninsbl Oct 2, 2022

ninsbl commented Oct 5, 2022

cwhite911 commented Nov 12, 2022

lucadelu commented Apr 27, 2023

veroandreo commented Jun 5, 2023

wenzeslaus commented Sep 19, 2023

neteler Sep 19, 2023

veroandreo Sep 19, 2023

ninsbl Sep 19, 2023

ninsbl commented Sep 19, 2023

neteler commented Sep 19, 2023

cwhite911 commented Sep 19, 2023

wenzeslaus commented Sep 19, 2023 •

edited

	gs.warning(_(f"Avaliable Collections: {avaliable_collections_ids}"))
	gs.warning(_("Avaliable Collections: {}").format(avaliable_collections_ids))

[WIP] t.stac.import: Add STAC API import functionality #802

Are you sure you want to change the base?

[WIP] t.stac.import: Add STAC API import functionality #802

Conversation

cwhite911 commented Sep 30, 2022

metzm commented Oct 4, 2022

veroandreo commented Oct 4, 2022 • edited by ninsbl

ninsbl left a comment

Choose a reason for hiding this comment

ninsbl Oct 2, 2022

Choose a reason for hiding this comment

ninsbl Oct 2, 2022

Choose a reason for hiding this comment

ninsbl commented Oct 5, 2022

cwhite911 commented Nov 12, 2022

lucadelu commented Apr 27, 2023

veroandreo commented Jun 5, 2023

wenzeslaus commented Sep 19, 2023

neteler Sep 19, 2023

Choose a reason for hiding this comment

veroandreo Sep 19, 2023

Choose a reason for hiding this comment

ninsbl Sep 19, 2023

Choose a reason for hiding this comment

ninsbl commented Sep 19, 2023

neteler commented Sep 19, 2023

cwhite911 commented Sep 19, 2023

wenzeslaus commented Sep 19, 2023 • edited

veroandreo commented Oct 4, 2022 •

edited by ninsbl

wenzeslaus commented Sep 19, 2023 •

edited