Series beginDateTime source in ODM2 timeseries DAO?

[miguelcleon]

@lsetiawan

I'm looking at the results from the following call, particularly the section below. Where does beginDateTime come from? I tried poking around in the code but I'm not finding it. The listed end date 2016-09-18T13:00:00 is correct but the first record for this time series is on 2016-07-07T10:00. This site was only active for a short time.

http://odm2admin.cuahsi.org/wofpy/odm2lczo/rest/1_1/GetSiteInfo?site=odm2lczo:RESDN

<variable>
<variableCode default="true" vocabulary="odm2lczo" variableID="15">Abs Pres</variableCode>
<variableName>Pressure, absolute</variableName>
<variableDescription>Water Pressure gauge</variableDescription>
<valueType>Observation</valueType>
<dataType>Average</dataType>
<generalCategory>Water quality</generalCategory>
<sampleMedium>Surface water</sampleMedium>
<unit>
<unitName>kilopascal</unitName>
<unitType>Pressure</unitType>
<unitAbbreviation>kPa</unitAbbreviation>
<unitCode>35</unitCode>
</unit>
<noDataValue>-6999.0000000000</noDataValue>
<timeScale/>
</variable>
<valueCount>5</valueCount>
<variableTimeInterval>
<beginDateTime>2014-03-10T09:00:00</beginDateTime>
<endDateTime>2016-09-18T13:00:00</endDateTime>

[lsetiawan]

It gets it from seriesResult here: https://github.com/ODM2/WOFpy/blob/master/wof/core_1_1.py#L779

[lsetiawan]

Probably it takes the first and last of the datetime in all your results values. If you don't think that is correct and it is not meeting what WaterML1_1 is supposed to do, please let me know and point me to the documentation of what is supposed to be there. Thanks.

[miguelcleon]

Yeah it would make sense if it where finding the first and last datetime in the result values for the series but in this case it is not finding the correct beginDateTime. I checked in the database. actually you have a copy resultid 16664 if you want to look.

[lsetiawan]

Okay, I will look into what's going on here. Thanks for letting me know.

[emiliom]

I'm looking into this too. Miguel, have you noticed that two of the variables do have a beginning in August 2017? On Oct 20, 2017 9:47 AM, "Landung Setiawan" <notifications@github.com> wrote: Okay, I will look into what's going on here. Thanks for letting me know. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#200 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAtUA3dfm4v3ZwWzrWKscnbnpWyrB49fks5suM6pgaJpZM4QA17F> .

[lsetiawan]

Hi All,

It seems like the values are grabbed from Action. https://github.com/ODM2/WOFpy/blob/master/wof/examples/flask/odm2/timeseries/sqlalch_odm2_models.py#L126

Here's what I get from debugging (All Action Instances)

BeginDateTime EndDateTime
2016-08-30 15:00:00 None
2014-03-10 09:00:00 None
2014-03-10 09:00:00 None
2014-03-10 09:00:00 None
2014-03-10 09:00:00 None
2016-08-30 15:00:00 None
2014-03-10 09:00:00 None
2014-03-10 09:00:00 None
2014-03-10 09:00:00 None

[lsetiawan]

Though the end date is not any of those values... hmm...

[emiliom]

Though the end date is not any of those values... hmm...

Well, see the logic starting on the next line:

if a_obj.EndDateTime is not None:
    self.EndDateTimeUTC = a_obj.EndDateTime.isoformat()
else:
    self.EndDateTimeUTC = r.tsrv_EndDateTime.isoformat()

A time series that's still ongoing (eg, sensor is still deployed) is supposed to have a NULL action EndDateTime; then, the response EndDateTime is grabbed from the actual time series result values. That's what the code is doing, so that part (end datetime) is working as intended.

Back to the BeginDateTime issue. @miguelcleon, I suspect what you're running into is not a WOFpy bug per se, but an ODM2 convention/expectation/best practice for time series results, that we (ODM2 gang) didn't make very obvious and you may not be following 😢. Namely, there is an action that corresponds to the current deployment of a sensor (result), and its BeginDateTime should be identical to the start of the time series. The alternative would be to query the time series result values to get the BeginDateTime, but we try to avoid doing that b/c it's a slow down on the WOFpy response. BUT, if you feel that would meet your needs better than the convention I've mentioned, you're free to customize your ODM2 time series DAO at exactly the line @lsetiawan pointed out! @lsetiawan can tell you if r (the results object referenced in r.tsrv_EndDateTime.isoformat()) already includes a r.tsrv_BeginDateTime; if it doesn't, the part of the DAO that queries the database would need to be tweaked. Even I can help you with that 😉

[lsetiawan]

Well, see the logic starting on the next line:

Haha yea.. I remember that logic now! We put that in place 😄 Thanks @emiliom

[emiliom]

A time series that's still ongoing (eg, sensor is still deployed) is supposed to have a NULL action EndDateTime; then, the response EndDateTime is grabbed from the actual time series result values. That's what the code is doing, so that part (end datetime) is working as intended.

I should add that the logic used in WOFpy can get tricky (ie, would need customization) if there's a chain of related actions during a sensor deployment. Or one can say, screw the attempt to minimize response times by leveraging actions, and just query the result values for both start and end datetimes to ensure accuracy!

[miguelcleon]

@emiliom Yeah, I don't have an action for each sensor deployment. In most cases I have a generic sensor deployment action that is just serving to link the sampling feature and method. The funny thing is I'm actually going through a fair amount of trouble storing start and end dates in some result extension property values in ODM2 Admin. 😢

[emiliom]

The funny thing is I'm actually going through a fair amount of trouble storing start and end dates in some result extension property values in ODM2 Admin

Well, the silver lining is that maybe you can customize your DAO to read from those result extension properties instead. I don't remember if odm2api already has a read service for result extension properties, but if it does, this may not be too much trouble (I can help next week, I think).

Then, let's discuss the broader issue of ODM2 conventions and challenges at the workshop, with a broader group! Maybe it's a good topic to discuss with everyone at the workshop, not just a small subset of people.

[miguelcleon]

Yes, looks like it does. Yes, discussing conventions and challenges does sound like a good idea. I'll look into customizing the DAO.