Questions about conversion from SOS to SensorThings

[haayman]

I'm trying to convert from SOS to SensorThings, but some things puzzle me.

Old situation:

• call http://data.smartemission.nl/sos52n/api/v1/stations?expanded=true
• for all station/timeseries I call http://data.smartemission.nl/sos52n/api/v1/timeseries//getData?timespan=PT48H/2018-1-29T13:40 to get the measurements for the last 48 hours

What's the SensorThings equivalent for the timeseries/getData call?

2. When I do this now (jan 29 2018 14:00) I get results up to the last hour, but when I look at http://data.smartemission.nl/gost/v1.0/Things?$expand=Datastreams/Observations the most recent measurement appears to be this morning 8:00. How is this different?

[justb4]

The SensorThings API is more navigation-oriented but has many other possibilities. As to start capturing some SmartEmission-specific use of STA and best-practices, I've started a Cookbook as a chapter in the documentation. Not yet finished but may give initial guidance:

http://smartplatform.readthedocs.io/en/latest/cookbook.html#sensorthings-api

The main three Entities to play with are Thing, Datastream and Observation. Often there is more than one way to get to the same data. It all depends on what your app is supposed to do, i.e. at which point in the user interaction what data needs to be available. Do you have a link to your app or its source?

Getting the last results (Observations) for a Station (Thing) is tricky. At least I did not find this could be effected using a single URL-query.

[justb4]

I think the scenario can be realized with STA as follows, see also the SE STA Cookbook:

Get All Stations with Locations

Long version, with Datastreams and Locations.

http://data.smartemission.nl/gost/v1.0/Things?$expand=Locations,Datastreams

Compact version, only attributes relevant for plotting icons on map.

http://data.smartemission.nl/gost/v1.0/Things?$expand=Locations($select=location)&$select=id,name)

To get all Stations: repeat the @iot.nextLink URL in response to go through all Pages.
I've increased the page size to 100 Entities (was 20).

Get Last Observations Per Station

Get the last Observations for single Station (device id = 131):

http://data.smartemission.nl/gost/v1.0/Things(131)?$expand=Datastreams/Observations($top=1)

Get the Observations for last 48 hours, for single Station (device id = 131): need to look further into. To get all Observations of a specific hour let’s say between 11:00 and 12:00 on January 29, 2018:

http://data.smartemission.nl/gost/v1.0/Observations?$filter=phenomenonTime gt ‘2018-01-29T11:00:00.000Z’ and phenomenonTime le ‘2018-01-29T12:00:00.000Z’&$select=result,phenomenonTime,parameters

Note that we are using GOST STA v0.5. In the upcoming v0.6 many fixes went in for queries. e.g. specifying ISO 8601 time periods may work.

[haayman]

I'm comparing the SOS and SensorThings API and am quite surprised about the differences. Aren't they supposed to render the same result?

http://data.smartemission.nl/sos52n/api/v1/stations

http://data.smartemission.nl/gost/v1.0/Locations

What I notice that the according to Locations almost all sensors are in Helmond, 1 is in Bilthoven and 4 sensors are at [-2.048575,-2.048575]

[justb4]

Yes, locations are supposed to be the same and I think they are. Have you read the Cookbook in particular the Paging feature, how objects from SE are mapped to STA Entities and how to query stations (via Things)? Querying on Locations is not the way. If I for example query for all stations (Things) in Zoetermeer (project_id eq '2001') I get coordinates in Zoetermeer like:

http://data.smartemission.nl/gost/v1.0/Things?$expand=Locations($select=location)&$select=id,properties,name&$filter=properties/project_id%20eq%20%272001%27

There are about 181 stations, many not active, many last seen in Helmond. Some give invalid GPS coordinates or no location at all. With STA: each time that sensors vals are updated, its Location is updated as well.

[haayman]

I know about the paging. I'm doing these 2 requests (so without filtering on project_id's, because I don't know about those):

http://data.smartemission.nl/gost/v1.0/Things?$expand=Datastreams,Locations
and http://data.smartemission.nl/gost/v1.0/Things?$expand=Datastreams,Locations&$top=100&$skip=100

The results of these are attached to this post.
What I also notice that the responses tell there are 182 results, but actually there are only 26 results aftere these 2 calls. Maybe the count is wrong due to the $expand parameter?

Things2.txt
Things1.txt

[justb4]

Confirmed: indeed if I request just the Things or Things with Locations I get all 182 Stations in two requests:
http://data.smartemission.nl/gost/v1.0/Things?$expand=Locations and then
http://data.smartemission.nl/gost/v1.0/Things?$expand=Locations&$top=100&$skip=100

It looks indeed like it counts all Entities returned when expanding both Locations and Datastreams. Will look further into...

[justb4]

According to the standard The count annotation represents the number of entities in the collection. : http://docs.opengeospatial.org/is/15-078r6/15-078r6.html#37

Looks like the @iot.nextLink is not properly set in relation to $iot.count. If I keep adding 100 to the $skip I get out all Things with some overlap, until it is empty at $skip=1700.

• http://data.smartemission.nl/gost/v1.0/Things?$expand=Datastreams,Locations&$top=100
• http://data.smartemission.nl/gost/v1.0/Things?$expand=Datastreams,Locations&$top=100&$skip=100
• http://data.smartemission.nl/gost/v1.0/Things?$expand=Datastreams,Locations&$top=100&$skip=200
• http://data.smartemission.nl/gost/v1.0/Things?$expand=Datastreams,Locations&$top=100&$skip=300
  .
  .
• http://data.smartemission.nl/gost/v1.0/Things?$expand=Datastreams,Locations&$top=100&$skip=1700 (empty)

Opened issue at GOST STA project: gost/server#146.

[haayman]

The workaround to keep increasing the $skip until no results are returned appears to work.
But this paging in combination with $expand creates a lot of problems/inefficiencies. See #90

[justb4]

@haayman @Hans-Nouwens
gost/server#146 is for a large part fixed and now rolled out on SE production server, see my comment on what works and still needs fixing there: gost/server#146 (comment)