Issue 2 retire rolling parameter in api decorator

documentation/api/change_log.rst

@@ -3,10 +3,21 @@
 Change log
 ==========
 
+v2.0 | 2021-02-23
+"""""""""""""""""
+
+- [**Breaking change**] Switched the interpretation of horizons to rolling horizons.
+- [**Breaking change**] Deprecated the use of ISO 8601 repeating time intervals to denote rolling horizons.
+- [**Breaking change**] Deprecated the automatic inference of horizons for *postMeterData*, *postPrognosis*, *postPriceData* and *postWeatherData* endpoints for API version below v2.0.
+- Introduced the "prior" parameter for *postMeterData*, *postPrognosis*, *postPriceData* and *postWeatherData* endpoints.
+- Changed the Introduction section:
+
+    - Rewrote the subsection on prognoses to explain the horizon and prior parameters
+
 v2.0 | 2020-11-14
 """""""""""""""""""
 
-- REST endpoints for managing assets: `/assets/` (GET, POST) and `/asset/<id>` (GET, PATCH, DELETE)
+- REST endpoints for managing assets: `/assets/` (GET, POST) and `/asset/<id>` (GET, PATCH, DELETE).
 
 v1.3-7 | 2020-12-16
 """""""""""""""""""

documentation/api/introduction.rst

@@ -307,12 +307,49 @@ For version 1 of the API, only univariate timeseries data is expected to be comm
 - "start" should be a timestamp on the hour or a multiple of 15 minutes thereafter, and
 - "duration" should be a multiple of 15 minutes.
 
+.. _beliefs:
+
+Beliefs
+^^^^^^^
+
+By regarding all time series data as beliefs that have been recorded at a certain time, data can be filtered accordingly.
+Some GET endpoints have two optional timing parameters to allow such filtering.
+The "prior" parameter (a timestamp) can be used to select beliefs recorded before some moment in time.
+It can be used to "time-travel" to see the state of information at some moment in the past.
+In addition, the "horizon" parameter (a duration) can be used to select beliefs recorded before some moment in time, relative to each event.
+For example, to filter out meter readings communicated within a day (denoted by a negative horizon) or forecasts created at least a day beforehand (denoted by a positive horizon).
+In addition to these two timing filters, beliefs can be filtered by their source (see :ref:`sources`).
+
+The two timing parameters follow the ISO 8601 standard and are interpreted as follows:
+
+- "horizon": recorded at least <duration> before the fact (indicated by a positive horizon), or at most <duration> after the fact (indicated by a negative horizon).
+- "prior": recorded prior to <timestamp>.
+
+For example:
+
+.. code-block:: json
+
+    {
+        "horizon": "PT6H",
+        "prior": "2020-08-01T17:00:00Z"
+    }
+
+These parameters denote that the data should have been recorded at least 6 hours before the fact (i.e. forecasts) and prior to 5 PM on August 1st 2020 (UTC).
+
 .. _prognoses:
 
 Prognoses
 ^^^^^^^^^
 
-When POSTing a prognosis, the message should state a time horizon, i.e. the duration between the time at which the prognosis was made and the time of realisation (commonly at the end of the prognosed time interval). The horizon can be stated explicitly by including a "horizon", consistent with the ISO 8601 standard, as follows:
+Some POST endpoints have two optional parameters to allow setting the time at which beliefs are recorded explicitly.
+This is useful to keep an accurate history of what was known at what time, especially for prognoses.
+If not used, |FLEXMEASURES_PLATFORM_NAME| will infer the prior from the arrival time of the message.
+
+The "prior" parameter (a timestamp) can be used to set a single time at which the entire prognosis was recorded.
+Alternatively, the "horizon" parameter (a duration) can be used to set the recording times relative to each prognosed event.
+In case both parameters are set, the earliest recording time is determined.
+
+The two timing parameters follow the ISO 8601 standard and are interpreted as follows:
 
 .. code-block:: json
 
@@ -324,11 +361,10 @@ When POSTing a prognosis, the message should state a time horizon, i.e. the dura
         ],
         "start": "2016-05-01T13:00:00Z",
         "duration": "PT45M",
-        "horizon": "PT6H"
+        "prior": "2016-05-01T07:45:00Z",
     }
 
-This message implies that the entire prognosis was made at 7:45 AM UTC, i.e. 6 hours before the end of the time interval.
-Alternatively, a rolling horizon can be stated as an ISO 8601 repeating time interval:
+This message implies that the entire prognosis was recorded at 7:45 AM UTC, i.e. 6 hours before the end of the entire time interval.
 
 .. code-block:: json
 
@@ -340,13 +376,14 @@ Alternatively, a rolling horizon can be stated as an ISO 8601 repeating time int
         ],
         "start": "2016-05-01T13:00:00Z",
         "duration": "PT45M",
-        "horizon": "R/PT6H"
+        "horizon": "PT6H"
     }
 
-Here, the number of repetitions and the repeat rule is omitted as it is implied by our notation for univariate timeseries (a complete representation of the "horizon" would have been "R3/PT6H/FREQ=MI;INTR=15").
-This message implies that the value for 1:00-1:15 PM was made at 7:15 AM, the value for 1:15-1:30 PM was made at 7:30 AM, and the value for 1:30-1:45 PM was made at 7:45 AM.
+This message implies that all prognosed values were recorded 6 hours in advance.
+That is, the value for 1:00-1:15 PM was made at 7:15 AM, the value for 1:15-1:30 PM was made at 7:30 AM, and the value for 1:30-1:45 PM was made at 7:45 AM.
 
-A "horizon" may be omitted, in which case the web service will infer the horizon from the arrival time of the message. Negative horizons may also be stated (breaking with the ISO 8601 standard) to indicate a prognosis about something that has already happened (i.e. after the fact, or simply *ex post*). For example, the following message implies that the entire prognosis was made at 1:55 PM UTC, 10 minutes after the fact:
+Negative horizons may also be stated (breaking with the ISO 8601 standard) to indicate a prognosis about something that has already happened (i.e. after the fact, or simply *ex post*).
+For example, the following message implies that all prognosed values were made 10 minutes after the fact:
 
 .. code-block:: json
 
@@ -361,37 +398,8 @@ A "horizon" may be omitted, in which case the web service will infer the horizon
         "horizon": "-PT10M"
     }
 
-For a rolling horizon indicating a prognosis 10 minutes after the start of each 15-minute interval, the "horizon" would have been "R/PT5M" since in fact only the last 5 minutes of each interval occurs before the fact (*ex ante*).
-That is, for ex-ante prognoses, the timeseries resolution (here 15 minutes) is included in the horizon, because the horizon is relative to the end of the timeseries.
-
-.. _beliefs:
-
-Beliefs
-^^^^^^^
-
-By regarding all time series data as beliefs that have been recorded at a certain time, data can be filtered accordingly.
-Some GET endpoints have two optional timing parameters to allow such filtering.
-The "prior" parameter (a timestamp) can be used to select beliefs recorded before some moment in time.
-It can be used to "time-travel" to see the state of information at some moment in the past.
-In addition, the "horizon" parameter (a duration) can be used to select beliefs recorded before some moment in time, relative to each event.
-For example, to filter out meter readings communicated within a day (denoted by a negative horizon) or forecasts created at least a day beforehand (denoted by a positive horizon).
-In addition to these two timing filters, beliefs can be filtered by their source (see :ref:`sources`).
-
-The two timing parameters follow the ISO 8601 standard and are interpreted as follows:
-
-- "horizon": recorded at least <duration> before the fact (indicated by a positive horizon), or at most <duration> after the fact (indicated by a negative horizon).
-- "prior": recorded prior to <timestamp>.
-
-For example:
-
-.. code-block:: json
-
-    {
-        "horizon": "PT6H",
-        "prior": "2020-08-01T17:00:00Z"
-    }
-
-These parameters denote that the data should have been recorded at least 6 hours before the fact (i.e. forecasts) and prior to 5 PM on August 1st 2020 (UTC).
+Note that, for a horizon indicating a prognosis 10 minutes after the *start* of each 15-minute interval, the "horizon" would have been "PT5M".
+This denotes that the prognosed interval has 5 minutes left to be concluded.
 
 .. _resolutions:
 

flexmeasures/api/common/utils/validators.py

@@ -307,7 +307,7 @@ def decorated_service(*args, **kwargs):
     return wrapper
 
 
-def optional_prior_accepted(ex_post: bool = False):
+def optional_prior_accepted(ex_post: bool = False, infer_missing: bool = True):
     """Decorator which specifies that a GET or POST request accepts an optional prior.
     It parses relevant form data and sets the "prior" keyword param.
 
@@ -316,6 +316,7 @@ def optional_prior_accepted(ex_post: bool = False):
     -   This results in the filter belief_time_window = (None, prior)
 
     Optionally, an ex_post flag can be passed to the decorator to indicate that only ex-post datetimes are allowed.
+    For POST requests, set infer_missing is True to have servers not in play mode derive a prior from the server time.
     """
 
     def wrapper(fn):
@@ -341,7 +342,14 @@ def decorated_service(*args, **kwargs):
                     if prior < knowledge_time:
                         extra_info = "Meter data can only be observed after the fact."
                         return invalid_horizon(extra_info)
+            elif (
+                infer_missing is True
+                and current_app.config.get("FLEXMEASURES_MODE", "") != "play"
+            ):
+                # A missing prior is inferred by the server (if not in play mode)
+                prior = server_now()
             else:
+                # Otherwise, a missing prior is fine (a horizon may still be inferred by the server)
                 prior = None
 
             kwargs["prior"] = prior
@@ -353,12 +361,14 @@ def decorated_service(*args, **kwargs):
 
 
 def optional_horizon_accepted(  # noqa C901
-    ex_post: bool = False, infer_missing: bool = True
+    ex_post: bool = False,
+    infer_missing: bool = True,
+    accept_repeating_interval: bool = False,
 ):
     """Decorator which specifies that a GET or POST request accepts an optional horizon.
     It parses relevant form data and sets the "horizon" keyword param.
-    For POST requests, the "rolling" keyword param is also set.
-    # todo: deprecate the rolling keyword param in favour of an optional "prior" parameter for POST requests
+    If accept_repeating_interval is True, the "rolling" keyword param is also set
+    (this was used for POST requests before v2.0)
 
     Interpretation for GET requests:
     -   Denotes "at least <horizon> before the fact (positive horizon),
@@ -380,8 +390,7 @@ def post_meter_data(horizon):
             return 'Meter data posted'
 
     If the message specifies a "horizon", it should be in accordance with the ISO 8601 standard.
-    If no "horizon" is specified for a POST request, it is determined by the server.
-    The play server uses 0 hours as a default horizon, while other servers derive the horizon from the server time.
+    For POST requests, set infer_missing is True to have servers in play mode use 0 hours as a default horizon.
     """
 
     def wrapper(fn):
@@ -405,43 +414,24 @@ def decorated_service(*args, **kwargs):
                     if horizon > timedelta(hours=0):
                         extra_info = "Meter data must have a zero or negative horizon to indicate observations after the fact."
                         return invalid_horizon(extra_info)
-            elif infer_missing is True:
-                # A missing horizon is only accepted if the server can infer it
-                if "start" in form and "duration" in form:
-                    start = parse_isodate_str(form["start"])
-                    duration = parse_duration(form["duration"], start)
-                    if not start:
-                        extra_info = "Cannot parse 'start' value."
-                        current_app.logger.warning(extra_info)
-                        return invalid_period(extra_info)
-                    if start.tzinfo is None:
-                        current_app.logger.warning(
-                            "Cannot parse timezone of 'start' value"
-                        )
-                        return invalid_timezone(
-                            "Start time should explicitly state a timezone."
-                        )
-                    if not duration:
-                        extra_info = "Cannot parse 'duration' value."
-                        current_app.logger.warning(extra_info)
-                        return invalid_period(extra_info)
-                    if current_app.config.get("FLEXMEASURES_MODE", "") == "play":
-                        horizon = timedelta(hours=0)
-                    else:
-                        horizon = start + duration - server_now()
-                    rolling = False
-                else:
-                    current_app.logger.warning(
-                        "Request missing both 'horizon', 'start' and 'duration'."
-                    )
-                    extra_info = "Specify a 'horizon' value, or 'start' and 'duration' values so that the horizon can be inferred."
+                elif rolling is True and accept_repeating_interval is False:
+                    extra_info = "The use of ISO 8601 repeating time intervals has been deprecated since API version 2.0."
                     return invalid_horizon(extra_info)
+            elif (
+                infer_missing is True
+                and current_app.config.get("FLEXMEASURES_MODE", "") == "play"
+            ):
+                # A missing horizon is set to zero for servers in play mode
+                horizon = timedelta(hours=0)
+            elif infer_missing is True and accept_repeating_interval is True:
+                extra_info = "Horizon inference deprecated for API versions below v2.0."
+                return invalid_horizon(extra_info)
             else:
-                # Otherwise, a missing horizon is fine
+                # Otherwise, a missing horizon is fine (a prior may still be inferred by the server)
                 horizon = None
 
             kwargs["horizon"] = horizon
-            if infer_missing is True:
+            if infer_missing is True and accept_repeating_interval is True:
                 kwargs["rolling"] = rolling
             return fn(*args, **kwargs)
 

flexmeasures/api/v1/implementations.py

@@ -48,8 +48,10 @@
 @units_accepted("power", "MW")
 @assets_required("connection")
 @optional_user_sources_accepted(default_source="MDC")
-@optional_horizon_accepted(ex_post=True, infer_missing=False)
-@optional_prior_accepted(ex_post=True)
+@optional_horizon_accepted(
+    ex_post=True, infer_missing=False, accept_repeating_interval=True
+)
+@optional_prior_accepted(ex_post=True, infer_missing=False)
 @period_required
 @get_data_downsampling_allowed("connection")
 @as_json
@@ -98,7 +100,7 @@ def get_meter_data_response(
 @units_accepted("power", "MW")
 @assets_required("connection")
 @values_required
-@optional_horizon_accepted(ex_post=True)
+@optional_horizon_accepted(ex_post=True, accept_repeating_interval=True)
 @period_required
 @post_data_checked_for_required_resolution("connection")
 @as_json

flexmeasures/api/v1/tests/utils.py

@@ -75,6 +75,7 @@ def message_for_post_meter_data(
         ],
         "start": "2015-01-01T00:00:00Z",
         "duration": duration_isoformat(timedelta(hours=1.5 * tile_n)),
+        "horizon": "PT0H",
         "unit": "MW",
     }
     if no_connection:

flexmeasures/api/v1_1/implementations.py

@@ -66,7 +66,7 @@ def get_connection_response():
 @type_accepted("PostPriceDataRequest")
 @units_accepted("price", "EUR/MWh", "KRW/kWh")
 @assets_required("market")
-@optional_horizon_accepted()
+@optional_horizon_accepted(accept_repeating_interval=True)
 @values_required
 @period_required
 @post_data_checked_for_required_resolution("market")
@@ -163,7 +163,7 @@ def post_price_data_response(
 @type_accepted("PostWeatherDataRequest")
 @unit_required
 @assets_required("sensor")
-@optional_horizon_accepted()
+@optional_horizon_accepted(accept_repeating_interval=True)
 @values_required
 @period_required
 @post_data_checked_for_required_resolution("sensor")
@@ -267,8 +267,8 @@ def post_weather_data_response(  # noqa: C901
 @units_accepted("power", "MW")
 @assets_required("connection")
 @optional_user_sources_accepted()
-@optional_horizon_accepted(infer_missing=False)
-@optional_prior_accepted()
+@optional_horizon_accepted(infer_missing=False, accept_repeating_interval=True)
+@optional_prior_accepted(infer_missing=False)
 @period_required
 @get_data_downsampling_allowed("connection")
 @as_json
@@ -309,7 +309,7 @@ def get_prognosis_response(
 @units_accepted("power", "MW")
 @assets_required("connection")
 @values_required
-@optional_horizon_accepted(ex_post=False)
+@optional_horizon_accepted(ex_post=False, accept_repeating_interval=True)
 @period_required
 @post_data_checked_for_required_resolution("connection")
 @as_json

flexmeasures/api/v2_0/implementations/__init__.py

@@ -0,0 +1 @@
+from . import assets, sensors, users  # noqa F401