Entity address scheme improvements

documentation/api/change_log.rst

@@ -3,27 +3,47 @@
 API change log
 ===============
 
+v2.0 | 2021-03-23
+"""""""""""""""""
+
+- [**Breaking change**] Switched the interpretation of horizons to rolling horizons.
+- [**Breaking change**] Deprecated the use of ISO 8601 repeating time intervals to denote rolling horizons.
+- [**Breaking change**] Deprecated the automatic inference of horizons for *postMeterData*, *postPrognosis*, *postPriceData* and *postWeatherData* endpoints for API version below v2.0.
+- Introduced the "prior" field for *postMeterData*, *postPrognosis*, *postPriceData* and *postWeatherData* endpoints.
+- Changed the Introduction section:
+
+    - Rewrote the subsection on prognoses to explain the horizon and prior fields.
+
+- Changed the Simulation section:
+
+    - Rewrote relevant examples using horizon and prior fields.
+
+v2.0 | 2021-02-19
+"""""""""""""""""""
+
+- REST endpoints for managing users: `/users/` (GET), `/user/<id>` (GET, PATCH) and `/user/<id>/password-reset` (PATCH).
+
 v2.0 | 2020-11-14
 """""""""""""""""""
 
-- REST endpoints for managing assets: `/assets/` (GET, POST) and `/asset/<id>` (GET, PATCH, DELETE)
+- REST endpoints for managing assets: `/assets/` (GET, POST) and `/asset/<id>` (GET, PATCH, DELETE).
 
 v1.3-7 | 2020-12-16
 """""""""""""""""""
 
 *Affects all versions since v1.0*.
 
-- Separated the dual purpose of the "horizon" parameter in the *getMeterData* and *getPrognosis* endpoints by introducing the "prior" parameter:
+- Separated the dual purpose of the "horizon" field in the *getMeterData* and *getPrognosis* endpoints by introducing the "prior" field:
 
-    - The "horizon" parameter in GET endpoints is now always interpreted as a rolling horizon, regardless of whether it is stated as an ISO 8601 repeating time interval.
-    - The *getMeterData* and *getPrognosis* endpoints now accept an optional "prior" parameter to select only data recorded before a certain ISO 8601 timestamp (replacing the unintuitive usage of the horizon field for specifying a latest time of belief).
+    - The "horizon" field in GET endpoints is now always interpreted as a rolling horizon, regardless of whether it is stated as an ISO 8601 repeating time interval.
+    - The *getMeterData* and *getPrognosis* endpoints now accept an optional "prior" field to select only data recorded before a certain ISO 8601 timestamp (replacing the unintuitive usage of the horizon field for specifying a latest time of belief).
 
 v1.3-6 | 2020-12-11
 """""""""""""""""""
 
 *Affects all versions since v1.0*.
 
-- The *getMeterData* and *getPrognosis* endpoints now return the INVALID_SOURCE status 400 response in case the optional "source" parameter is used and no relevant sources can be found.
+- The *getMeterData* and *getPrognosis* endpoints now return the INVALID_SOURCE status 400 response in case the optional "source" field is used and no relevant sources can be found.
 
 v1.3-5 | 2020-10-29
 """""""""""""""""""
@@ -32,7 +52,7 @@ v1.3-5 | 2020-10-29
 
 - Endpoints to POST meter data will now check incoming data to see if the required asset's resolution is being used ― upsampling is done if possible.
   These endpoints can now return the REQUIRED_INFO_MISSING status 400 response.
-- Endpoints to GET meter data will return data in the asset's resolution ― downsampling to the "resolution" parameter is done if possible.
+- Endpoints to GET meter data will return data in the asset's resolution ― downsampling to the "resolution" field is done if possible.
 - As they need to determine the asset, all of the mentioned POST and GET endpoints can now return the UNRECOGNIZED_ASSET status 4000 response.
 
 v1.3-4 | 2020-06-18

documentation/api/introduction.rst

@@ -4,7 +4,7 @@ Introduction
 ============
 
 This document details the Application Programming Interface (API) of the FlexMeasures web service. The API supports user automation for flexibility valorisation in the energy sector, both in a live setting and for the purpose of simulating scenarios. The web service adheres to the concepts and terminology used in the Universal Smart Energy Framework (USEF).
-We assume in this document that the FlexMeasures instance you want to connect to is hosted at https://company.flexmeasures.io. 
+We assume in this document that the FlexMeasures instance you want to connect to is hosted at https://company.flexmeasures.io.
 
 
 New versions of the API are released on:
@@ -13,7 +13,7 @@ New versions of the API are released on:
 
     https://company.flexmeasures.io/api
 
-A list of services offered by (a version of) the FlexMeasures web service can be obtained by sending a *getService* request. An optional parameter "access" can be used to specify a user role for which to obtain only the relevant services.
+A list of services offered by (a version of) the FlexMeasures web service can be obtained by sending a *getService* request. An optional field "access" can be used to specify a user role for which to obtain only the relevant services.
 
 **Example request**
 
@@ -307,12 +307,49 @@ For version 1 of the API, only univariate timeseries data is expected to be comm
 - "start" should be a timestamp on the hour or a multiple of 15 minutes thereafter, and
 - "duration" should be a multiple of 15 minutes.
 
+.. _beliefs:
+
+Beliefs
+^^^^^^^
+
+By regarding all time series data as beliefs that have been recorded at a certain time, data can be filtered accordingly.
+Some GET endpoints have two optional timing fields to allow such filtering.
+The "prior" field (a timestamp) can be used to select beliefs recorded before some moment in time.
+It can be used to "time-travel" to see the state of information at some moment in the past.
+In addition, the "horizon" field (a duration) can be used to select beliefs recorded before some moment in time, relative to each event.
+For example, to filter out meter readings communicated within a day (denoted by a negative horizon) or forecasts created at least a day beforehand (denoted by a positive horizon).
+In addition to these two timing filters, beliefs can be filtered by their source (see :ref:`sources`).
+
+The two timing fields follow the ISO 8601 standard and are interpreted as follows:
+
+- "horizon": recorded at least <duration> before the fact (indicated by a positive horizon), or at most <duration> after the fact (indicated by a negative horizon).
+- "prior": recorded prior to <timestamp>.
+
+For example:
+
+.. code-block:: json
+
+    {
+        "horizon": "PT6H",
+        "prior": "2020-08-01T17:00:00Z"
+    }
+
+These fields denote that the data should have been recorded at least 6 hours before the fact (i.e. forecasts) and prior to 5 PM on August 1st 2020 (UTC).
+
 .. _prognoses:
 
 Prognoses
 ^^^^^^^^^
 
-When POSTing a prognosis, the message should state a time horizon, i.e. the duration between the time at which the prognosis was made and the time of realisation (commonly at the end of the prognosed time interval). The horizon can be stated explicitly by including a "horizon", consistent with the ISO 8601 standard, as follows:
+Some POST endpoints have two optional fields to allow setting the time at which beliefs are recorded explicitly.
+This is useful to keep an accurate history of what was known at what time, especially for prognoses.
+If not used, |FLEXMEASURES_PLATFORM_NAME| will infer the prior from the arrival time of the message.
+
+The "prior" field (a timestamp) can be used to set a single time at which the entire prognosis was recorded.
+Alternatively, the "horizon" field (a duration) can be used to set the recording times relative to each prognosed event.
+In case both fields are set, the earliest possible recording time is determined and recorded for each prognosed event.
+
+The two timing fields follow the ISO 8601 standard and are interpreted as follows:
 
 .. code-block:: json
 
@@ -324,11 +361,10 @@ When POSTing a prognosis, the message should state a time horizon, i.e. the dura
         ],
         "start": "2016-05-01T13:00:00Z",
         "duration": "PT45M",
-        "horizon": "PT6H"
+        "prior": "2016-05-01T07:45:00Z",
     }
 
-This message implies that the entire prognosis was made at 7:45 AM UTC, i.e. 6 hours before the end of the time interval.
-Alternatively, a rolling horizon can be stated as an ISO 8601 repeating time interval:
+This message implies that the entire prognosis was recorded at 7:45 AM UTC, i.e. 6 hours before the end of the entire time interval.
 
 .. code-block:: json
 
@@ -340,13 +376,14 @@ Alternatively, a rolling horizon can be stated as an ISO 8601 repeating time int
         ],
         "start": "2016-05-01T13:00:00Z",
         "duration": "PT45M",
-        "horizon": "R/PT6H"
+        "horizon": "PT6H"
     }
 
-Here, the number of repetitions and the repeat rule is omitted as it is implied by our notation for univariate timeseries (a complete representation of the "horizon" would have been "R3/PT6H/FREQ=MI;INTR=15").
-This message implies that the value for 1:00-1:15 PM was made at 7:15 AM, the value for 1:15-1:30 PM was made at 7:30 AM, and the value for 1:30-1:45 PM was made at 7:45 AM.
+This message implies that all prognosed values were recorded 6 hours in advance.
+That is, the value for 1:00-1:15 PM was made at 7:15 AM, the value for 1:15-1:30 PM was made at 7:30 AM, and the value for 1:30-1:45 PM was made at 7:45 AM.
 
-A "horizon" may be omitted, in which case the web service will infer the horizon from the arrival time of the message. Negative horizons may also be stated (breaking with the ISO 8601 standard) to indicate a prognosis about something that has already happened (i.e. after the fact, or simply *ex post*). For example, the following message implies that the entire prognosis was made at 1:55 PM UTC, 10 minutes after the fact:
+Negative horizons may also be stated (breaking with the ISO 8601 standard) to indicate a prognosis about something that has already happened (i.e. after the fact, or simply *ex post*).
+For example, the following message implies that all prognosed values were made 10 minutes after the fact:
 
 .. code-block:: json
 
@@ -361,37 +398,8 @@ A "horizon" may be omitted, in which case the web service will infer the horizon
         "horizon": "-PT10M"
     }
 
-For a rolling horizon indicating a prognosis 10 minutes after the start of each 15-minute interval, the "horizon" would have been "R/PT5M" since in fact only the last 5 minutes of each interval occurs before the fact (*ex ante*).
-That is, for ex-ante prognoses, the timeseries resolution (here 15 minutes) is included in the horizon, because the horizon is relative to the end of the timeseries.
-
-.. _beliefs:
-
-Beliefs
-^^^^^^^
-
-By regarding all time series data as beliefs that have been recorded at a certain time, data can be filtered accordingly.
-Some GET endpoints have two optional timing parameters to allow such filtering.
-The "prior" parameter (a timestamp) can be used to select beliefs recorded before some moment in time.
-It can be used to "time-travel" to see the state of information at some moment in the past.
-In addition, the "horizon" parameter (a duration) can be used to select beliefs recorded before some moment in time, relative to each event.
-For example, to filter out meter readings communicated within a day (denoted by a negative horizon) or forecasts created at least a day beforehand (denoted by a positive horizon).
-In addition to these two timing filters, beliefs can be filtered by their source (see :ref:`sources`).
-
-The two timing parameters follow the ISO 8601 standard and are interpreted as follows:
-
-- "horizon": recorded at least <duration> before the fact (indicated by a positive horizon), or at most <duration> after the fact (indicated by a negative horizon).
-- "prior": recorded prior to <timestamp>.
-
-For example:
-
-.. code-block:: json
-
-    {
-        "horizon": "PT6H",
-        "prior": "2020-08-01T17:00:00Z"
-    }
-
-These parameters denote that the data should have been recorded at least 6 hours before the fact (i.e. forecasts) and prior to 5 PM on August 1st 2020 (UTC).
+Note that, for a horizon indicating a prognosis 10 minutes after the *start* of each 15-minute interval, the "horizon" would have been "PT5M".
+This denotes that the prognosed interval has 5 minutes left to be concluded.
 
 .. _resolutions:
 

documentation/api/simulation.rst

@@ -86,7 +86,8 @@ Weather data can be posted for the following three types of weather sensors:
 
 The sensor type is part of the unique entity address for each sensor, together with the sensor's latitude and longitude.
 
-This "PostWeatherDataRequest" message posts temperature forecasts for 15-minute intervals between 3.00pm and 4.30pm for a weather sensor located at latitude 33.4843866 and longitude 126.477859. The forecasts were made at noon.
+This "PostWeatherDataRequest" message posts temperature forecasts for 15-minute intervals between 3.00pm and 4.30pm for a weather sensor located at latitude 33.4843866 and longitude 126.477859.
+The forecasts were made at noon.
 
 .. code-block:: json
 
@@ -103,20 +104,22 @@ This "PostWeatherDataRequest" message posts temperature forecasts for 15-minute
             ],
             "start": "2015-01-01T15:00:00+09:00",
             "duration": "PT1H30M",
-            "horizon": "PT4H30M",
+            "prior": "2015-01-01T12:00:00+09:00",
             "unit": "°C"
         }
 
 Observations vs forecasts
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 
-To post an observation rather than a forecast, simply set the horizon to "PT0H".
+To post an observation rather than a forecast, simply set the prior to the moment at which the observations were made, e.g. at "2015-01-01T16:30:00+09:00".
 This denotes that the observation was made exactly after realisation of this list of temperature readings, i.e. at 4.30pm.
 
-Alternatively, to indicate that each individual observation was made directly after the end of its 15-minute interval (i.e. at 3.15pm, 3.30pm and so on), set the horizon to "R/PT0H".
+Alternatively, to indicate that each individual observation was made directly after the end of its 15-minute interval (i.e. at 3.15pm, 3.30pm and so on), set a horizon to "PT0H" instead of a prior.
 
 Finally, delays in reading out sensor data can be simulated by setting the horizon field to a negative value.
-For example, a horizon of "-PT1H" would denote that this list of temperature readings was observed one hour after the fact (i.e. at 5.30pm).
+For example, a horizon of "-PT1H" would denote that each temperature reading was observed one hour after the fact (i.e. at 4.15pm, 4.30 pm and so on).
+
+See :ref:`prognoses` for more information regarding the prior and horizon fields.
 
 
 Posting price data
@@ -339,7 +342,7 @@ This example requests a prognosis with a rolling horizon of 6 hours before reali
         "connection": "ea1.2018-06.io.flexmeasures.company:1:1",
         "start": "2015-01-01T00:00:00+00:00",
         "duration": "PT24H",
-        "horizon": "R/PT6H",
+        "horizon": "PT6H",
         "resolution": "PT15M",
         "unit": "MW"
     }

documentation/changelog.rst

@@ -6,11 +6,16 @@ FlexMeasures Changelog
 v0.2.4 | March XX, 2021
 ===========================
 
+New features
+------------
+* Optionally setting recording time when posting data [see `PR #41 <http://www.github.com/SeitaBV/flexmeasures/pull/41>`_]
+
 Bugfixes
 --------
 * Show screenshots in documentation and add some missing content [see `PR #60 <http://www.github.com/SeitaBV/flexmeasures/pull/60>`_]
 * Documentation listed 2.0 API endpoints twice [see `PR #59 <http://www.github.com/SeitaBV/flexmeasures/pull/59>`_]
 * User page did not list number of assets correctly [see `PR #64 <http://www.github.com/SeitaBV/flexmeasures/pull/64>`_]
+* Missing *postPrognosis* endpoint for >1.0 API blueprints [part of `PR #41 <http://www.github.com/SeitaBV/flexmeasures/pull/41>`_]
 
 Infrastructure/Support
 ----------------------

flexmeasures/api/common/schemas/sensors.py

@@ -0,0 +1,89 @@
+from typing import Union
+
+from marshmallow import fields
+
+from flexmeasures.api import FMValidationError
+from flexmeasures.api.common.utils.api_utils import get_weather_sensor_by
+from flexmeasures.utils.entity_address_utils import (
+    parse_entity_address,
+    EntityAddressException,
+)
+from flexmeasures.data.models.assets import Asset
+from flexmeasures.data.models.markets import Market
+from flexmeasures.data.models.weather import WeatherSensor
+from flexmeasures.data.models.time_series import Sensor
+
+
+class EntityAddressValidationError(FMValidationError):
+    status = "INVALID_DOMAIN"  # USEF error status
+
+
+class SensorField(fields.Str):
+    """Field that deserializes to a Sensor, Asset, Market or WeatherSensor
+    and serializes back to an entity address (string)."""
+
+    def __init__(
+        self,
+        entity_type: str,
+        *args,
+        **kwargs,
+    ):
+        """
+        :param entity_type: "sensor", "connection", "market" or "weather_sensor"
+        """
+        self.entity_type = entity_type
+        super().__init__(*args, **kwargs)
+
+    def _deserialize(
+        self, value, attr, obj, **kwargs
+    ) -> Union[Sensor, Asset, Market, WeatherSensor]:
+        """Deserialize to a Sensor, Asset, Market or WeatherSensor."""
+        try:
+            ea = parse_entity_address(value, self.entity_type)
+            if self.entity_type == "sensor":
+                sensor = Sensor.query.filter(Sensor.id == ea["sensor_id"]).one_or_none()
+                if sensor is not None:
+                    return sensor
+                else:
+                    raise EntityAddressValidationError(
+                        f"Sensor with entity address {value} doesn't exist."
+                    )
+            elif self.entity_type == "connection":
+                asset = Asset.query.filter(Asset.id == ea["asset_id"]).one_or_none()
+                if asset is not None:
+                    return asset
+                else:
+                    raise EntityAddressValidationError(
+                        f"Asset with entity address {value} doesn't exist."
+                    )
+            elif self.entity_type == "market":
+                market = Market.query.filter(
+                    Market.name == ea["market_name"]
+                ).one_or_none()
+                if market is not None:
+                    return market
+                else:
+                    raise EntityAddressValidationError(
+                        f"Market with entity address {value} doesn't exist."
+                    )
+            elif self.entity_type == "weather_sensor":
+                weather_sensor = get_weather_sensor_by(
+                    ea["weather_sensor_type_name"], ea["latitude"], ea["longitude"]
+                )
+                if weather_sensor is not None and isinstance(
+                    weather_sensor, WeatherSensor
+                ):
+                    return weather_sensor
+                else:
+                    raise EntityAddressValidationError(
+                        f"Weather sensor with entity address {value} doesn't exist."
+                    )
+        except EntityAddressException as eae:
+            raise EntityAddressValidationError(str(eae))
+        return NotImplemented
+
+    def _serialize(
+        self, value: Union[Sensor, Asset, Market, WeatherSensor], attr, data, **kwargs
+    ):
+        """Serialize to an entity address."""
+        return value.entity_address