Issue 2 retire rolling parameter in api decorator

documentation/api/change_log.rst

@@ -3,27 +3,47 @@
 Change log
 ==========
 
+v2.0 | 2021-02-23
+"""""""""""""""""
+
+- [**Breaking change**] Switched the interpretation of horizons to rolling horizons.
+- [**Breaking change**] Deprecated the use of ISO 8601 repeating time intervals to denote rolling horizons.
+- [**Breaking change**] Deprecated the automatic inference of horizons for *postMeterData*, *postPrognosis*, *postPriceData* and *postWeatherData* endpoints for API version below v2.0.
+- Introduced the "prior" field for *postMeterData*, *postPrognosis*, *postPriceData* and *postWeatherData* endpoints.
+- Changed the Introduction section:
+
+    - Rewrote the subsection on prognoses to explain the horizon and prior fields.
+
+- Changed the Simulation section:
+
+    - Rewrote relevant examples using horizon and prior fields.
+
+v2.0 | 2021-02-19
+"""""""""""""""""""
+
+- REST endpoints for managing users: `/users/` (GET), `/user/<id>` (GET, PATCH) and `/user/<id>/password-reset` (PATCH).
+
 v2.0 | 2020-11-14
 """""""""""""""""""
 
-- REST endpoints for managing assets: `/assets/` (GET, POST) and `/asset/<id>` (GET, PATCH, DELETE)
+- REST endpoints for managing assets: `/assets/` (GET, POST) and `/asset/<id>` (GET, PATCH, DELETE).
 
 v1.3-7 | 2020-12-16
 """""""""""""""""""
 
 *Affects all versions since v1.0*.
 
-- Separated the dual purpose of the "horizon" parameter in the *getMeterData* and *getPrognosis* endpoints by introducing the "prior" parameter:
+- Separated the dual purpose of the "horizon" field in the *getMeterData* and *getPrognosis* endpoints by introducing the "prior" field:
 
-    - The "horizon" parameter in GET endpoints is now always interpreted as a rolling horizon, regardless of whether it is stated as an ISO 8601 repeating time interval.
-    - The *getMeterData* and *getPrognosis* endpoints now accept an optional "prior" parameter to select only data recorded before a certain ISO 8601 timestamp (replacing the unintuitive usage of the horizon field for specifying a latest time of belief).
+    - The "horizon" field in GET endpoints is now always interpreted as a rolling horizon, regardless of whether it is stated as an ISO 8601 repeating time interval.
+    - The *getMeterData* and *getPrognosis* endpoints now accept an optional "prior" field to select only data recorded before a certain ISO 8601 timestamp (replacing the unintuitive usage of the horizon field for specifying a latest time of belief).
 
 v1.3-6 | 2020-12-11
 """""""""""""""""""
 
 *Affects all versions since v1.0*.
 
-- The *getMeterData* and *getPrognosis* endpoints now return the INVALID_SOURCE status 400 response in case the optional "source" parameter is used and no relevant sources can be found.
+- The *getMeterData* and *getPrognosis* endpoints now return the INVALID_SOURCE status 400 response in case the optional "source" field is used and no relevant sources can be found.
 
 v1.3-5 | 2020-10-29
 """""""""""""""""""
@@ -32,7 +52,7 @@ v1.3-5 | 2020-10-29
 
 - Endpoints to POST meter data will now check incoming data to see if the required asset's resolution is being used ― upsampling is done if possible.
   These endpoints can now return the REQUIRED_INFO_MISSING status 400 response.
-- Endpoints to GET meter data will return data in the asset's resolution ― downsampling to the "resolution" parameter is done if possible.
+- Endpoints to GET meter data will return data in the asset's resolution ― downsampling to the "resolution" field is done if possible.
 - As they need to determine the asset, all of the mentioned POST and GET endpoints can now return the UNRECOGNIZED_ASSET status 4000 response.
 
 v1.3-4 | 2020-06-18

documentation/api/introduction.rst

@@ -13,7 +13,7 @@ New versions of the API are released on:
 
     https://company.flexmeasures.io/api
 
-A list of services offered by (a version of) the |FLEXMEASURES_PLATFORM_NAME| web service can be obtained by sending a *getService* request. An optional parameter "access" can be used to specify a user role for which to obtain only the relevant services.
+A list of services offered by (a version of) the |FLEXMEASURES_PLATFORM_NAME| web service can be obtained by sending a *getService* request. An optional field "access" can be used to specify a user role for which to obtain only the relevant services.
 
 **Example request**
 
@@ -307,12 +307,49 @@ For version 1 of the API, only univariate timeseries data is expected to be comm
 - "start" should be a timestamp on the hour or a multiple of 15 minutes thereafter, and
 - "duration" should be a multiple of 15 minutes.
 
+.. _beliefs:
+
+Beliefs
+^^^^^^^
+
+By regarding all time series data as beliefs that have been recorded at a certain time, data can be filtered accordingly.
+Some GET endpoints have two optional timing fields to allow such filtering.
+The "prior" field (a timestamp) can be used to select beliefs recorded before some moment in time.
+It can be used to "time-travel" to see the state of information at some moment in the past.
+In addition, the "horizon" field (a duration) can be used to select beliefs recorded before some moment in time, relative to each event.
+For example, to filter out meter readings communicated within a day (denoted by a negative horizon) or forecasts created at least a day beforehand (denoted by a positive horizon).
+In addition to these two timing filters, beliefs can be filtered by their source (see :ref:`sources`).
+
+The two timing fields follow the ISO 8601 standard and are interpreted as follows:
+
+- "horizon": recorded at least <duration> before the fact (indicated by a positive horizon), or at most <duration> after the fact (indicated by a negative horizon).
+- "prior": recorded prior to <timestamp>.
+
+For example:
+
+.. code-block:: json
+
+    {
+        "horizon": "PT6H",
+        "prior": "2020-08-01T17:00:00Z"
+    }
+
+These fields denote that the data should have been recorded at least 6 hours before the fact (i.e. forecasts) and prior to 5 PM on August 1st 2020 (UTC).
+
 .. _prognoses:
 
 Prognoses
 ^^^^^^^^^
 
-When POSTing a prognosis, the message should state a time horizon, i.e. the duration between the time at which the prognosis was made and the time of realisation (commonly at the end of the prognosed time interval). The horizon can be stated explicitly by including a "horizon", consistent with the ISO 8601 standard, as follows:
+Some POST endpoints have two optional fields to allow setting the time at which beliefs are recorded explicitly.
+This is useful to keep an accurate history of what was known at what time, especially for prognoses.
+If not used, |FLEXMEASURES_PLATFORM_NAME| will infer the prior from the arrival time of the message.
+
+The "prior" field (a timestamp) can be used to set a single time at which the entire prognosis was recorded.
+Alternatively, the "horizon" field (a duration) can be used to set the recording times relative to each prognosed event.
+In case both fields are set, the earliest recording time is determined.
+
+The two timing fields follow the ISO 8601 standard and are interpreted as follows:
 
 .. code-block:: json
 
@@ -324,11 +361,10 @@ When POSTing a prognosis, the message should state a time horizon, i.e. the dura
         ],
         "start": "2016-05-01T13:00:00Z",
         "duration": "PT45M",
-        "horizon": "PT6H"
+        "prior": "2016-05-01T07:45:00Z",
     }
 
-This message implies that the entire prognosis was made at 7:45 AM UTC, i.e. 6 hours before the end of the time interval.
-Alternatively, a rolling horizon can be stated as an ISO 8601 repeating time interval:
+This message implies that the entire prognosis was recorded at 7:45 AM UTC, i.e. 6 hours before the end of the entire time interval.
 
 .. code-block:: json
 
@@ -340,13 +376,14 @@ Alternatively, a rolling horizon can be stated as an ISO 8601 repeating time int
         ],
         "start": "2016-05-01T13:00:00Z",
         "duration": "PT45M",
-        "horizon": "R/PT6H"
+        "horizon": "PT6H"
     }
 
-Here, the number of repetitions and the repeat rule is omitted as it is implied by our notation for univariate timeseries (a complete representation of the "horizon" would have been "R3/PT6H/FREQ=MI;INTR=15").
-This message implies that the value for 1:00-1:15 PM was made at 7:15 AM, the value for 1:15-1:30 PM was made at 7:30 AM, and the value for 1:30-1:45 PM was made at 7:45 AM.
+This message implies that all prognosed values were recorded 6 hours in advance.
+That is, the value for 1:00-1:15 PM was made at 7:15 AM, the value for 1:15-1:30 PM was made at 7:30 AM, and the value for 1:30-1:45 PM was made at 7:45 AM.
 
-A "horizon" may be omitted, in which case the web service will infer the horizon from the arrival time of the message. Negative horizons may also be stated (breaking with the ISO 8601 standard) to indicate a prognosis about something that has already happened (i.e. after the fact, or simply *ex post*). For example, the following message implies that the entire prognosis was made at 1:55 PM UTC, 10 minutes after the fact:
+Negative horizons may also be stated (breaking with the ISO 8601 standard) to indicate a prognosis about something that has already happened (i.e. after the fact, or simply *ex post*).
+For example, the following message implies that all prognosed values were made 10 minutes after the fact:
 
 .. code-block:: json
 
@@ -361,37 +398,8 @@ A "horizon" may be omitted, in which case the web service will infer the horizon
         "horizon": "-PT10M"
     }
 
-For a rolling horizon indicating a prognosis 10 minutes after the start of each 15-minute interval, the "horizon" would have been "R/PT5M" since in fact only the last 5 minutes of each interval occurs before the fact (*ex ante*).
-That is, for ex-ante prognoses, the timeseries resolution (here 15 minutes) is included in the horizon, because the horizon is relative to the end of the timeseries.
-
-.. _beliefs:
-
-Beliefs
-^^^^^^^
-
-By regarding all time series data as beliefs that have been recorded at a certain time, data can be filtered accordingly.
-Some GET endpoints have two optional timing parameters to allow such filtering.
-The "prior" parameter (a timestamp) can be used to select beliefs recorded before some moment in time.
-It can be used to "time-travel" to see the state of information at some moment in the past.
-In addition, the "horizon" parameter (a duration) can be used to select beliefs recorded before some moment in time, relative to each event.
-For example, to filter out meter readings communicated within a day (denoted by a negative horizon) or forecasts created at least a day beforehand (denoted by a positive horizon).
-In addition to these two timing filters, beliefs can be filtered by their source (see :ref:`sources`).
-
-The two timing parameters follow the ISO 8601 standard and are interpreted as follows:
-
-- "horizon": recorded at least <duration> before the fact (indicated by a positive horizon), or at most <duration> after the fact (indicated by a negative horizon).
-- "prior": recorded prior to <timestamp>.
-
-For example:
-
-.. code-block:: json
-
-    {
-        "horizon": "PT6H",
-        "prior": "2020-08-01T17:00:00Z"
-    }
-
-These parameters denote that the data should have been recorded at least 6 hours before the fact (i.e. forecasts) and prior to 5 PM on August 1st 2020 (UTC).
+Note that, for a horizon indicating a prognosis 10 minutes after the *start* of each 15-minute interval, the "horizon" would have been "PT5M".
+This denotes that the prognosed interval has 5 minutes left to be concluded.
 
 .. _resolutions:
 

documentation/api/simulation.rst

@@ -86,7 +86,8 @@ Weather data can be posted for the following three types of weather sensors:
 
 The sensor type is part of the unique entity address for each sensor, together with the sensor's latitude and longitude.
 
-This "PostWeatherDataRequest" message posts temperature forecasts for 15-minute intervals between 3.00pm and 4.30pm for a weather sensor located at latitude 33.4843866 and longitude 126.477859. The forecasts were made at noon.
+This "PostWeatherDataRequest" message posts temperature forecasts for 15-minute intervals between 3.00pm and 4.30pm for a weather sensor located at latitude 33.4843866 and longitude 126.477859.
+The forecasts were made at noon.
 
 .. code-block:: json
 
@@ -103,20 +104,22 @@ This "PostWeatherDataRequest" message posts temperature forecasts for 15-minute
             ],
             "start": "2015-01-01T15:00:00+09:00",
             "duration": "PT1H30M",
-            "horizon": "PT4H30M",
+            "prior": "2015-01-01T12:00:00+09:00",
             "unit": "°C"
         }
 
 Observations vs forecasts
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 
-To post an observation rather than a forecast, simply set the horizon to "PT0H".
+To post an observation rather than a forecast, simply set the prior to the moment at which the observations were made, e.g. at "2015-01-01T16:30:00+09:00".
 This denotes that the observation was made exactly after realisation of this list of temperature readings, i.e. at 4.30pm.
 
-Alternatively, to indicate that each individual observation was made directly after the end of its 15-minute interval (i.e. at 3.15pm, 3.30pm and so on), set the horizon to "R/PT0H".
+Alternatively, to indicate that each individual observation was made directly after the end of its 15-minute interval (i.e. at 3.15pm, 3.30pm and so on), set a horizon to "PT0H" instead of a prior.
 
 Finally, delays in reading out sensor data can be simulated by setting the horizon field to a negative value.
-For example, a horizon of "-PT1H" would denote that this list of temperature readings was observed one hour after the fact (i.e. at 5.30pm).
+For example, a horizon of "-PT1H" would denote that each temperature reading was observed one hour after the fact (i.e. at 4.15pm, 4.30 pm and so on).
+
+See :ref:`prognoses` for more information regarding the prior and horizon fields.
 
 
 Posting price data
@@ -339,7 +342,7 @@ This example requests a prognosis with a rolling horizon of 6 hours before reali
         "connection": "ea1.2018-06.io.flexmeasures.company:1:1",
         "start": "2015-01-01T00:00:00+00:00",
         "duration": "PT24H",
-        "horizon": "R/PT6H",
+        "horizon": "PT6H",
         "resolution": "PT15M",
         "unit": "MW"
     }

flexmeasures/api/common/utils/api_utils.py

@@ -6,15 +6,23 @@
 from flask import current_app
 from inflection import pluralize
 from numpy import array
+from rq.job import Job
+from sqlalchemy.exc import IntegrityError
 
 from flexmeasures.data import db
-from flexmeasures.data.models.assets import Asset
-from flexmeasures.data.models.markets import Market
+from flexmeasures.data.models.assets import Asset, Power
+from flexmeasures.data.models.markets import Market, Price
 from flexmeasures.data.models.data_sources import DataSource
-from flexmeasures.data.models.weather import WeatherSensor
+from flexmeasures.data.models.weather import WeatherSensor, Weather
 from flexmeasures.data.models.user import User
+from flexmeasures.data.utils import save_to_session
 from flexmeasures.utils.entity_address_utils import parse_entity_address
-from flexmeasures.api.common.responses import unrecognized_sensor
+from flexmeasures.api.common.responses import (
+    unrecognized_sensor,
+    ResponseTuple,
+    request_processed,
+    already_received_and_successfully_processed,
+)
 
 
 def list_access(service_listing, service_name):
@@ -337,7 +345,9 @@ def get_weather_sensor_by(
     return weather_sensor
 
 
-def get_generic_asset(asset_descriptor, entity_type):
+def get_generic_asset(
+    asset_descriptor, entity_type
+) -> Union[Asset, Market, WeatherSensor, None]:
     """
     Get a generic asset from form information
     # TODO: After refactoring, unify 3 generic_asset cases -> 1 sensor case
@@ -356,3 +366,64 @@ def get_generic_asset(asset_descriptor, entity_type):
             ea["longitude"],
         )
     return None
+
+
+def save_to_db(
+    timed_values: List[Union[Power, Price, Weather]], forecasting_jobs: List[Job]
+) -> ResponseTuple:
+    """Put the timed values into the database and create forecasting jobs.
+
+    Data can only be replaced on servers in play mode.
+
+    :param timed_values: list of Power, Price or Weather values to be saved
+    :param forecasting_jobs: list of forecasting Jobs for redis queues.
+    :returns: ResponseTuple
+    """
+    current_app.logger.info("SAVING TO DB AND QUEUEING...")
+    try:
+        save_to_session(timed_values)
+        db.session.flush()
+        [current_app.queues["forecasting"].enqueue_job(job) for job in forecasting_jobs]
+        db.session.commit()
+        return request_processed()
+    except IntegrityError as e:
+        current_app.logger.warning(e)
+        db.session.rollback()
+
+        # Allow data to be replaced only in play mode
+        if current_app.config.get("FLEXMEASURES_MODE", "") == "play":
+            save_to_session(timed_values, overwrite=True)
+            [
+                current_app.queues["forecasting"].enqueue_job(job)
+                for job in forecasting_jobs
+            ]
+            db.session.commit()
+            return request_processed()
+        else:
+            return already_received_and_successfully_processed()
+
+
+def determine_belief_horizons(event_values, start, resolution, horizon, prior, sensor):
+    """Determine belief horizons given a horizon, prior, or both,
+    and taking into account the sensor's knowledge horizon function."""
+    event_starts = [start + j * resolution for j in range(len(event_values))]
+    if horizon is not None and prior is not None:
+        belief_horizons_from_horizon = [horizon] * len(event_values)
+        belief_horizons_from_prior = [
+            event_start - prior - sensor.knowledge_horizon(event_start)
+            for event_start in event_starts
+        ]
+        belief_horizons = [
+            max(a, b)
+            for a, b in zip(belief_horizons_from_horizon, belief_horizons_from_prior)
+        ]
+    elif horizon is not None:
+        belief_horizons = [horizon] * len(event_values)
+    elif prior is not None:
+        belief_horizons = [
+            event_start - prior - sensor.knowledge_horizon(event_start)
+            for event_start in event_starts
+        ]
+    else:
+        raise ValueError("Missing horizon or prior.")
+    return event_starts, event_values, belief_horizons