[sensorthings] Support feature expansion

[nyalldawson]

This change allows SensorThings entities to be expanded to contain their related child feature attributes, exposing the relational SensorThings model as a traditional "flat" GIS-friendly table structure.

Eg when selecting Location entities, you can now opt to expand to "Things > Datastreams > Observations". This would result in multiple "stacked" point location features, one corresponding to each observation, with the attributes for each point feature containing the location, thing, datastream and observation attributes.

Best used combined with some extent, feature limit, or custom filter option, as this can otherwise result in very heavy
requests to the backend service! There's also an option to limit then number of child features returned when expanding, and we default to a very conservative amount here in order to reduce load on services. (I.e. users need to explicitly "opt in" to fetch large amounts of features)

Here's an example of a feature returned when the expansion option out to "Location > Thing > Datastream" is used. Note how the "Thing" and "Thing_Datastream" attributes are appended on to the base "Location" attributes:

Fixes #56805

[qgis-bot]

@nyalldawson
This pull request has been tagged as requiring documentation.

A documentation ticket will be opened at https://github.com/qgis/QGIS-Documentation when this PR is merged.

Please update the description (not the comments) with helpful description and screenshot to help the work from documentors.
Also, any commit having [needs-doc] or [Needs Documentation] in will see its message pushed to the issue, so please be as verbose as you can.

Thank you!

[qgis-bot]

@nyalldawson

This pull request has been tagged for the changelog.

• The description will be harvested so please provide a "nearly-ready" text for the final changelog
• If possible, add a nice illustration of the feature. Only the first one in the description will be harvested (GIF accepted as well)
• If you can, it's better to give credits to your sponsor, see below for different formats.

You can edit the description.

Format available for credits

• Funded by NAME
• Funded by URL
• Funded by NAME URL
• Sponsored by NAME
• Sponsored by URL
• Sponsored by NAME URL

Thank you!

[github-actions]

🪟 Windows builds ready!

Windows builds of this PR are available for testing here. Debug symbols for this build are available here.

(Built from commit da6bb1f)

[nyalldawson]

(not for merge at this stage -- after discussion with @rduivenvoorde we need a sort order option to accompany this)

[rduivenvoorde]

@nyalldawson, not so much viewing from my use-cases, but more with QGIS/GIS in mind: should the 'Expand to' rules not mostly (try to) expand to objects in which every object will get a 'geometry'? Instead of, for example: Observations only be available now with 'No Geometry'?

I have to try to craft some of the 'queries' first, but I reckoned that the ideal situation would be some query/question like: Get me all (recent x) Observations (from a given Datastream) in this extent and show them on my MapCanvas (and I can use Temporal Controller to filter/view them)? This would mean that this (Datastream)-Observations should at least contain the expansion till the Location of their Thing? So the Geometry type would be Point then?

I think(!) the responsibility to, for example, request all Datastreams ( to be able to show all Observations of a Quantity and Unit (ObservedProperty) of interest ), will probably be in a plugin (using core func)? So the plugin can then request all Observations from selected Datastreams (using core func).
I do not think requesting (with no geometry) DataStreams or Observations is interesting if I cannot show them on a map?

From https://github.com/FraunhoferIOSB/FROST-Server/wiki/Example-queries,
all things, with their current Locations and Datastreams, and for those Datastreams the ObservedProperty and the last 2(!) Observations:

# All things, with their current Locations and Datastreams, and for those Datastreams the ObservedProperty and the last 2(!) Observations:
https://airquality-frost.k8s.ilt-dmz.iosb.fraunhofer.de/v1.1/Things
  ?$select=name,description,@iot.id
  &$expand=
    Locations
      ($select=name,description,location,@iot.id)
    ,Datastreams
      ($select=name,description,@iot.id
      ;$expand=
        Sensor
          ($select=name,description,@iot.id)
        ,ObservedProperty
          ($select=name,description,@iot.id)
        ,Observations
          ($select=result,phenomenonTime,@iot.id
          ;$orderby=phenomenonTime%20desc
          ;$top=2
          )
      )

clickable: https://airquality-frost.k8s.ilt-dmz.iosb.fraunhofer.de/v1.1/Things?$count=true&$select=name,description,@iot.id&$expand=Locations($select=name,description,location,@iot.id),Datastreams($select=name,description,@iot.id;$expand=Sensor($select=name,description,@iot.id),ObservedProperty($select=name,description,@iot.id),Observations($select=result,phenomenonTime,@iot.id;$orderby=phenomenonTime%20desc;$top=2))

This probably should also be possible 'the other way around', so request Observations (of a certain Datastream (== type)) and expand Thing/Location so every Observation has a geometry?

OR am I going over the top anyway? Because this makes the queries (and handling of the results) too complex?

But then would be my question: do we have bindings (for plugin authors) to craft such queries (and retrieve/parse the data theirselves?

[rduivenvoorde]

Another example

All Datastreams (in given bbox, though not sure if st_contains is still in v1.1...), then expand the Thing/Location to be able to show them on a map

https://airquality-frost.k8s.ilt-dmz.iosb.fraunhofer.de/v1.1/Datastreams?$count=true&$expand=Thing/Locations&$filter=st_contains(geography%27POLYGON%20((4.031%2052.001,%203.088%2051.277,%204.322%2051.133,%204.678%2051.747,%204.031%2052.001))%27,%20Thing/Locations/location)

[nyalldawson]

Ok I've reworked this to allow for more flexible configuration of expansion targets no. Instead of the previous combobox and limit widget, there's now button to open a dialog to configure expansions:

Clicking this shows a dialog allowing users to control exactly how expansions should be made, including per-expansion limits and ordering settings: (potentially this could be expanded in future to also expose per expansion query filters!)

This should address the "find me just the most recent observation for each location" type use case (ie that's what the configuration above would do)

[nyalldawson]

@rduivenvoorde

not so much viewing from my use-cases, but more with QGIS/GIS in mind: should the 'Expand to' rules not mostly (try to) expand to objects in which every object will get a 'geometry'? Instead of, for example: Observations only be available now with 'No Geometry'?

My thinking is that the user must always select the entity with the geometry as the base entity type. So eg if they want observations, the base entity type is Location and then they expand through Things, DataStreams, to Observations.

[rduivenvoorde]

@nyalldawson Question: can you maybe give some hints on how to use the provider to fetch data (as that is how Luca et al. will be going to use it in the plugin(s))?

@ghtmtt I've build and did some test:

-1-

Observation, expanding to Datastream:
(same goes for FeaturesOfInterest expanding to Datastream)

Then there seems to be 'Observations'-table in the layer manager, but if you open the table (to fetch data), you see this query:

https://airquality-frost.k8s.ilt-dmz.iosb.fraunhofer.de/v1.1/Observations?$top=100&$count=false&$expand=Datastreams($top=100)

While (trial and error) what works is (note the 'Datastream' instead of 'DatastreamS':

https://airquality-frost.k8s.ilt-dmz.iosb.fraunhofer.de/v1.1/Observations?$top=100&$count=false&$expand=Datastream($top=100)

-2-

Another test, Thing expanding to Datastream just works (!):

https://airquality-frost.k8s.ilt-dmz.iosb.fraunhofer.de/v1.1/Things?$top=100&$count=false&$expand=Datastreams($orderby=id;$top=100)

Probably because one Thing can have several datastreams, but one Observation has always one datastream (looking at the diagram, I also read datastream vs datastreams)

-3-

My thinking is that the user must always select the entity with the geometry as the base entity type. So eg if they want >observations, the base entity type is Location and then they expand through Things, DataStreams, to Observations.

Is fine, a difficulty here (@hylkevds mentioned this in the chat during the meeting), is that you will get All datastreams of all Things at that moment, and thus a mixed set of Observations (as in different units/quantities/params).

Which could be fine!
But I think (but that would be one of the use cases) that I'm used to a layer/table with just one Quantity (only beta-radiation for example). But maybe that is then something to go through further filtering in QGIS itself?

Or (and this was mentioned by Luca ( no githandle available :-) ), we probably want do some more filtering (adding the filter clause into the queries to filter on 'properties' or :

• you might want to be able to only fetch the (latest) Observations from Datastreams with a given ObservedProperty (only SO2 sensors for example)?
• But showing these on the Map and thus expanding all untill the location of the Datastreams/Thing/Location seems a little overkill?

Example of all Things with SO2 sensor datastreams and the latest Observation:

https://airquality-frost.k8s.ilt-dmz.iosb.fraunhofer.de/v1.1/Things?$count=true&$select=name,description,@iot.id&$expand=Locations($select=name,description,location,@iot.id),Datastreams($select=name,description,@iot.id;$filter=ObservedProperty/name%20eq%20%27SO2%27;$expand=Sensor($select=name,description,@iot.id),ObservedProperty($select=name,description,@iot.id),Observations($select=result,phenomenonTime,@iot.id;$orderby=phenomenonTime%20desc;$top=1))

The crux is to create such (to me pretty complex) 'queries' in such a way that the result is a 'feature' with some geometry and value to chow in the maps

Another crafted example (just to get the Locations to be the root item). All locations expanding the Datastreams (selecting only the SO2 streams) via Things, and then showing the most recent Observation:

https://airquality-frost.k8s.ilt-dmz.iosb.fraunhofer.de/v1.1/Locations?$count=true&$select=name,location&$expand=Things/Datastreams($select=name,description,@iot.id;$filter=ObservedProperty/name%20eq%20%27SO2%27;$expand=Sensor($select=name,description,@iot.id),ObservedProperty($select=name,description,@iot.id),Observations($select=result,phenomenonTime,@iot.id;$orderby=phenomenonTime%20desc;$top=1))

Trying to minimize output:

https://airquality-frost.k8s.ilt-dmz.iosb.fraunhofer.de/v1.1/Locations?$count=true&$select=name,location&$expand=Things/Datastreams($select=name,description,@iot.id;$filter=ObservedProperty/name%20eq%20%27SO2%27;$expand=Observations($select=result,phenomenonTime,@iot.id;$orderby=phenomenonTime%20desc;$top=1))

-4- another observation: it is not so easy to clean the 'expand' path in the dialog? Changing Object does not clean the (then invalid) path. Sometimes it is cleaned (seemingly random?).

[nyalldawson]

@rduivenvoorde

@nyalldawson Question: can you maybe give some hints on how to use the provider to fetch data (as that is how Luca et al. will be going to use it in the plugin(s))?

Do you mean from a raw api point of view? If so, then I'd suggest either adding layers from data source manager and viewing the resultant layer source, OR checking out how we construct layer source URIs from the unit tests here: https://github.com/qgis/QGIS/blob/master/tests/src/python/test_provider_sensorthings.py#L3675

Observation, expanding to Datastream:
snip
While (trial and error) what works is (note the 'Datastream' instead of 'DatastreamS':

Wow, I do NOT like this part of the SensorThings specification! It turns out the the url query required for expansions depends on the actual relationship type between each expansion and its parent. This isn't nice at all for clients, and results in a lot more complex code 😭 . Anyway, I've addressed this in the latest commit.

(same goes for FeaturesOfInterest expanding to Datastream)

While we create the correct query string for this one now, it kills the frost server 🙃 The query we now send is of the form:

https://airquality-frost.k8s.ilt-dmz.iosb.fraunhofer.de/v1.1/FeaturesOfInterest?$top=200&$count=false&$expand=Observations($top=2;$expand=Datastream($top=2))

Which just endlessly hangs until the request times out. I doubt there's anything we can do from QGIS' side to handle this.

Or (and this was mentioned by Luca ( no githandle available :-) ), we probably want do some more filtering (adding the filter clause into the queries to filter on 'properties' or :
snip

I'm going to have to defer to @ghtmtt here -- my understanding is that this is considered out of scope for the current project and needs to wait till a further round of funding.

another observation: it is not so easy to clean the 'expand' path in the dialog? Changing Object does not clean the (then invalid) path. Sometimes it is cleaned (seemingly random?).

Try the latest version, I've added some tweaks

[hylkevds]

Observation, expanding to Datastream:
snip
While (trial and error) what works is (note the 'Datastream' instead of 'DatastreamS':

Wow, I do NOT like this part of the SensorThings specification! It turns out the the url query required for expansions depends on the actual relationship type between each expansion and its parent. This isn't nice at all for clients, and results in a lot more complex code 😭 . Anyway, I've addressed this in the latest commit.

The name of the relation has no technical relation to the name of the entity type on the other side of that relation in OData, so that name is based on the semantics of the relation. An Observation has one (1) Datastream, so the name of the relation is Datastream. A Thing has multiple Datastreams, so the name of the relation is Datastreams.
In v2 there will be a relation from the Datastream EntityType to the Feature EntityType called UltimateFeatureOfInterest, while there will be a relation from Observation to Feature called ProximateFeatureOfInterest... (diagram)

Two EntityTypes can even have multiple relations, which will happen in the Sampling extension in STA v2 between Sampling and Feature.

Better get it correct now, or there will be a serious issue when V2 comes along :)

(same goes for FeaturesOfInterest expanding to Datastream)

While we create the correct query string for this one now, it kills the frost server 🙃 The query we now send is of the form:

https://airquality-frost.k8s.ilt-dmz.iosb.fraunhofer.de/v1.1/FeaturesOfInterest?$top=200&$count=false&$expand=Observations($top=2;$expand=Datastream($top=2))

Which just endlessly hangs until the request times out. I doubt there's anything we can do from QGIS' side to handle this.

Missing index :)
It is, however, a meaningless query. You'd be getting two random Observations for each Feature, without any control over which ObservedProperty they may have. Could be the same, or different. And for the next Feature you may be getting different ObservedProperties... Expanding Observations on Features in STAv1 only makes sense if the Feature is a Sample with a limited set of Observations. That will be improved a lot in V2, by linking Features to Datastreams directly (for UltimateFeatures), and by adding FeatureTypes so you can find which Features are Samples.

[rduivenvoorde]

@ghtmtt we really have to discus use-cases here, because in current setup it is (I think) too easy to fire "meaningless queries' as @hylkevds says.

Some quick ideas/use-cases:

• the sta-plugin one: fetch all Locations (I think these should be Things with a Location), show them on the map, and from there fetch the Datastreams first (of ONE Thing/Location) and then fetch the Observations of that Datastream (to be able to show all data of one Datastream (== of one quantity/unit) in a table or graph

• my use-case 1: fetch all latest observations of all Datastreams (or a selection based on the ObservedProperty) and show these on their Thing/Location. Then in second phase (eg when a value is over a threshold), inspect the full timeserie of that one Datastream

• my usecase 2: Given a selected (set of) Datastream(s), from a driving sensor-vehicle, get all FeaturesOfInterest of those Datastreams and plot those (including the data) on a map, so you will have a view of the path of the moving sensor AND it's values on every part of that path

I'm aware these are all specific use-cases, but I think we should take this kind of use cases into mind when developing this provider.
I think users will (in the case of using STA in QGIS) always in the end want to see values or values and locations on the map (my usecases), or use the map as an interface to inspect a sensor (the STA-plugin usecase). Others please follow up.

[nyalldawson]

@rduivenvoorde

I think a good practical solution to that would be if we added "preset" configurations which prepopulate all the layer settings (including expansions).

Users could pick from these, or use them as starting points for configuration, or just configure from scratch themselves if no presets match their needs.

I'm shortly heading away for a week and a bit, so I'll catch up on this discussion when I'm back online!

[hylkevds]

Two more use cases:

1. Show all Things, with their current location, and their HistoricalLocations (with a time-slider)
2. Show all Things, with their current location, coloured by the result of the latest Observation of a given ObservedProperty

[lucagiovannini]

Dear all, here I am to join this very interesting discussion...which, in my opinion, is also quite pivotal to design a usable solution (both for data provider and for the data analysis plugin).

I think we are facing 2 main issues:

• design a user interface that can work both for simple and progressively complex queries, allows for the widest range of use cases, while keeping the user experience simple and clear (this is valid for both data provider and plugin);
• think of a way for the plugin to interact with the data provider (to fetch the data it needs for the analysis) that is at the same time: easy to implement for both provider and plugin, does not require complex/tedious actions for the user;

I thought about and drafted (on paper!) some ideas (and mock-ups too :) ) on how to tackle those issues and I will post them here as soon as I manage to transcribe them. I have my two young kids home from school for Easter and I can't manage right now ...but I will manage at most by early next week. I trust and I hope you'll be on some king of Easter holiday too! :)

[lucagiovannini]

Done! Sent by email!

[github-actions]

The QGIS project highly values your contribution and would love to see this work merged! Unfortunately this PR has not had any activity in the last 14 days and is being automatically marked as "stale". If you think this pull request should be merged, please check

• that all unit tests are passing

• that all comments by reviewers have been addressed

• that there is enough information for reviewers, in particular

  • link to any issues which this pull request fixes

  • add a description of workflows which this pull request fixes

  • add screenshots if applicable

• that you have written unit tests where possible
  In case you should have any uncertainty, please leave a comment and we will be happy to help you proceed with this pull request.
  If there is no further activity on this pull request, it will be closed in a week.

[github-actions]

While we hate to see this happen, this PR has been automatically closed because it has not had any activity in the last 21 days. If this pull request should be reconsidered, please follow the guidelines in the previous comment and reopen this pull request. Or, if you have any further questions, just ask! We love to help, and if there's anything the QGIS project can do to help push this PR forward please let us know how we can assist.

[github-actions]

The QGIS project highly values your contribution and would love to see this work merged! Unfortunately this PR has not had any activity in the last 14 days and is being automatically marked as "stale". If you think this pull request should be merged, please check

• that all unit tests are passing

• that all comments by reviewers have been addressed

• that there is enough information for reviewers, in particular

  • link to any issues which this pull request fixes

  • add a description of workflows which this pull request fixes

  • add screenshots if applicable

• that you have written unit tests where possible
  In case you should have any uncertainty, please leave a comment and we will be happy to help you proceed with this pull request.
  If there is no further activity on this pull request, it will be closed in a week.

[nyalldawson]

Merging after extensive user testing. Follow ups will come in separate prs

[qgis-bot]

@nyalldawson
A documentation ticket has been opened at qgis/QGIS-Documentation#9107
It is your responsibility to visit this ticket and add as much detail as possible for the documentation team to correctly document this change.
Thank you!