FlexMeasures · nhoening · Apr 21, 2021 · Apr 9, 2021 · Apr 9, 2021 · Apr 13, 2021
diff --git a/documentation/api/change_log.rst b/documentation/api/change_log.rst
@@ -3,12 +3,11 @@
 API change log
 ===============
 
-v2.0 | 2021-04-02
+v2.0-2 | 2021-04-02
 """""""""""""""""
 
 - [**Breaking change**] Switched the interpretation of horizons to rolling horizons.
 - [**Breaking change**] Deprecated the use of ISO 8601 repeating time intervals to denote rolling horizons.
-- [**Breaking change**] Deprecated the automatic inference of horizons for *postMeterData*, *postPrognosis*, *postPriceData* and *postWeatherData* endpoints for API version below v2.0.
 - Introduced the "prior" field for *postMeterData*, *postPrognosis*, *postPriceData* and *postWeatherData* endpoints.
 - Changed the Introduction section:
 
@@ -18,16 +17,31 @@ v2.0 | 2021-04-02
 
     - Rewrote relevant examples using horizon and prior fields.
 
-v2.0 | 2021-02-19
+v2.0-1 | 2021-02-19
 """""""""""""""""""
 
 - REST endpoints for managing users: `/users/` (GET), `/user/<id>` (GET, PATCH) and `/user/<id>/password-reset` (PATCH).
 
-v2.0 | 2020-11-14
+v2.0-0 | 2020-11-14
 """""""""""""""""""
 
 - REST endpoints for managing assets: `/assets/` (GET, POST) and `/asset/<id>` (GET, PATCH, DELETE).
 
+
+v1.3.9 | 2021-04-XX
+"""""""""""""""""
+
+*Affects all versions since v1.0*.
+
+- Fixed regression by partially reverting the breaking change of v1.3-8: Re-instantiated automatic inference of horizons for Post requests for API versions below v2.0, but changed to inference policy: now inferring the data was recorded **right after each event** took place (leading to a zero horizon for each data point) rather than **after the last event** took place (which led to a different horizon for each data point); the latter had been the inference policy before v1.3-8.
+
+v1.3-8 | 2020-04-02
+"""""""""""""""""""
+
+*Affects all versions since v1.0*.
+
+- [**Breaking change**, partially reverted in v1.3-9] Deprecated the automatic inference of horizons for *postMeterData*, *postPrognosis*, *postPriceData* and *postWeatherData* endpoints for API version below v2.0.
+
 v1.3-7 | 2020-12-16
 """""""""""""""""""
 
@@ -155,10 +169,7 @@ v1.1-2 | 2018-08-15
 - Added the *postPriceData* endpoint
 - Added a description of the *postPrognosis* endpoint in the Aggregator section
 - Added a description of the *postPriceData* endpoint in the Aggregator and Supplier sections
-
-.. ifconfig:: FLEXMEASURES_MODE == "play"
-
-    - Added the *restoreData* endpoint
+- Added the *restoreData* endpoint for servers in play mode
 
 v1.1-1 | 2018-08-06
 """""""""""""""""""

diff --git a/documentation/api/introduction.rst b/documentation/api/introduction.rst
@@ -257,7 +257,9 @@ In case of a single group of connections, the message may be flattened to:
 Timeseries
 ^^^^^^^^^^
 
-Timestamps and durations are consistent with the ISO 8601 standard. All timestamps in requests to the API must be timezone-aware. The timezone indication "Z" indicates a zero offset from UTC. Additionally, we use the following shorthand for sequential values within a time interval:
+Timestamps and durations are consistent with the ISO 8601 standard. The resolution of the data is implicit, see :ref:`resolutions`.
+
+All timestamps in requests to the API must be timezone-aware. The timezone indication "Z" indicates a zero offset from UTC. Additionally, we use the following shorthand for sequential values within a time interval:
 
 .. code-block:: json
 
@@ -405,8 +407,9 @@ This denotes that the prognosed interval has 5 minutes left to be concluded.
 Resolutions
 ^^^^^^^^^^^
 
-Specifying a resolution is redundant for POST requests that contain both "values" and a "duration".
-Also, posted data is checked against the required resolution of the assets which are posted to.
+Specifying a resolution is redundant for POST requests that contain both "values" and a "duration" ― FlexMeasures computes the resolution by dividing the duration by the number of values.
+
+When POSTing data, FlexMeasures checks this computed resolution against the required resolution of the assets which are posted to. If these can't be matched (through upsampling), an error will occur.
 
 GET requests (such as *getMeterData*) return data in the resolution which the sensor is configured for.
 A "resolution" may be specified explicitly to obtain the data in downsampled form, 

diff --git a/documentation/changelog.rst b/documentation/changelog.rst
@@ -18,7 +18,8 @@ Bugfixes
 Infrastructure / Support
 ----------------------
 * Updated dependencies, including Flask-Security-Too [see `PR #82 <http://www.github.com/SeitaBV/flexmeasures/pull/82>`_]
-* Integration with `timely beliefs <https://github.com/SeitaBV/timely-beliefs>`_ lib: Sensor data as TimedBeliefs [see `PR #79 <http://www.github.com/SeitaBV/flexmeasures/pull/79>`_]
+* Improved documentation after user feedback [see `PR #97 <http://www.github.com/SeitaBV/flexmeasures/pull/97>`_]
+* Begin experimental integration with `timely beliefs <https://github.com/SeitaBV/timely-beliefs>`_ lib: Sensor data as TimedBeliefs [see `PR #79 <http://www.github.com/SeitaBV/flexmeasures/pull/79>`_]
 * Add sensors with CLI command currently meant for developers only [see `PR #83 <https://github.com/SeitaBV/flexmeasures/pull/83>`_]
 * Add data (beliefs about sensor events) with CLI command currently meant for developers only [see `PR #85 <https://github.com/SeitaBV/flexmeasures/pull/85>`_]
 

diff --git a/documentation/configuration.rst b/documentation/configuration.rst
@@ -26,11 +26,14 @@ Level above which log messages are added to the log file. See the ``logging`` pa
 
 Default: ``logging.WARNING``
 
+
+.. _modes-config:
+
 FLEXMEASURES_MODE
 ^^^^^^^^^^^^^^^^^
 
 The mode in which FlexMeasures is being run, e.g. "demo" or "play".
-This is used to turn on certain extra behaviours.
+This is used to turn on certain extra behaviours, see :ref:`modes-dev` for details.
 
 Default: ``""``
 
@@ -369,13 +372,17 @@ Default: ``None``
 Demonstrations
 --------------
 
+.. _demo-credentials-config:
+
 FLEXMEASURES_PUBLIC_DEMO_CREDENTIALS
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 When ``FLEXMEASURES_MODE=demo``\ , this can hold login credentials (demo user email and password, e.g. ``("demo at seita.nl", "flexdemo")``\ ), so anyone can log in and try out the platform.
 
 Default: ``None``
 
+.. _demo-year-config:
+
 FLEXMEASURES_DEMO_YEAR
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

diff --git a/documentation/dev/data.rst b/documentation/dev/data.rst
@@ -100,36 +100,35 @@ Or, from within Postgres console:
    CREATE DATABASE flexmeasures_test WITH OWNER = flexmeasures_test;
 
 
-Log in as the postgres superuser and connect to your newly-created database:
+Finally, test if you can log in as the flexmeasures user:
 
 .. code-block:: bash
 
-   sudo -u postgres psql
+   psql -U flexmeasures --password -h 127.0.0.1 -d flexmeasures
 
 .. code-block:: sql
 
-   \connect flexmeasures
+   \q
+
 
+Add Postgres Extensions to your database(s)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
+To find the nearest sensors, FlexMeasures needs some extra POstgres support. 
 Add the following extensions while logged in as the postgres superuser:
 
+.. code-block:: bash
+
+   sudo -u postgres psql
+
 .. code-block:: sql
 
+   \connect flexmeasures
    CREATE EXTENSION cube;
    CREATE EXTENSION earthdistance;
 
 
-Connect to the ``flexmeasures_test`` database and repeat creating these extensions there. Then ``exit``.
-
-Finally, try logging in as the flexmeasures user once:
-
-.. code-block:: bash
-
-   psql -U flexmeasures --password -h 127.0.0.1 -d flexmeasures
-
-.. code-block:: sql
-
-   \q
+If you have it, connect to the ``flexmeasures_test`` database and repeat creating these extensions there. Then ``exit``.
 
 
 Configure FlexMeasures app for that database

diff --git a/documentation/dev/modes.rst b/documentation/dev/modes.rst
@@ -0,0 +1,41 @@
+.. _modes-dev:
+
+Modes
+============
+
+FlexMeasures can be run in specific modes (see the :ref:`modes-config` config setting).
+This is useful for certain special situations. Two are supported out of the box and we document here 
+how FlexMeasures behaves differently in these modes.
+
+Demo
+-------
+
+In this mode, the server is assumed to be used as a demonstration tool. Most of the following adaptations therefore happen in the UI. 
+
+- [Data] Demo data is often from an older source, and it's a hassle to change the year to the current year. FlexMeasures allows to set :ref:`demo-year-config` and when in ``demo`` mode, the current year will be translated to that year in the background.   
+- [UI] Logged-in users can view queues on the demo server (usually only admins can do that)
+- [UI] Demo servers often display login credentials, so visitors can try out functionality. Use the :ref:`demo-credentials-config` config setting to do this.
+- [UI] The dashboard shows all non-empty asset groups, instead of only the ones for the current user.
+- [UI] The analytics page mocks confidence intervals around power, price and weather data, so that the demo data doesn't need to have them. 
+- [UI] The portfolio page mocks flexibility numbers and a mocked control action.
+
+Play
+------
+
+In this mode, the server is assumed to be used to run simulations.
+
+Big features
+^^^^^^^^^^^^^
+
+- [Data] Allows overwriting existing data when saving data to the database.
+- [API] The inferred recording time of incoming data is immediately after the event took place, rather than the actual time at which the server received the data.
+- [API] Posting price or weather data does not trigger forecasting jobs.
+- [API] The ``restoreData`` endpoint is registered, enabling database resets through the API.
+- [API] When posting weather data for a new location, a new weather sensor is automatically created, instead of returning the nearest available weather sensor to post data to.
+
+Small features
+^^^^^^^^^^^^^^^
+
+- [API] Posted UDI events are not enforced to be consecutive.
+- [API] Names in ``GetConnectionResponse`` are the connections' unique database names rather than their display names (this feature is planned to be deprecated).
+- [UI] The dashboard plot showing the latest power value is not enforced to lie in the past (in case of simulating future values).
diff --git a/documentation/dev/plugins.rst b/documentation/dev/plugins.rst
@@ -76,4 +76,21 @@ All else that is needed for this showcase (not shown here) is ``<some_folder>/ou
 
 
 
-.. note:: Plugin views can also be added to the FlexMeasures UI menu ― just name them in the config setting :ref:`menu-config`.
+.. note:: Plugin views can also be added to the FlexMeasures UI menu ― just name them in the config setting :ref:`menu-config`.
+
+
+Using other files in your plugin
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Say you want to include other Python files in your plugin, importing them in your ``__init__.py`` file.
+This can be done if you put the plugin path on the import path. Do it like this in your ``__init__.py``:
+
+.. code-block:: python
+
+    import os
+    import sys
+
+    HERE = os.path.dirname(os.path.abspath(__file__))
+    sys.path.insert(0, HERE)
+
+    from my_other_file import my_function
diff --git a/documentation/getting-started.rst b/documentation/getting-started.rst
@@ -17,6 +17,7 @@ Install dependencies and the ``flexmeasures`` platform itself:
 
    pip install flexmeasures
 
+.. note:: With newer Python versions and Windows, some smaller dependencies (e.g. ``tables`` or ``rq-win``) might cause issues as support is often slower. You might overcome this with a little research, by `installing from wheels <http://www.pytables.org/usersguide/installation.html#prerequisitesbininst>`_ or `from the repo <https://github.com/michaelbrooks/rq-win#installation-and-use>`_, respectively.
 
 
 Make a secret key for sessions and password salts
@@ -217,7 +218,7 @@ To collect weather measurements and forecasts from the DarkSky API, there is a t
 
    flexmeasures add external-weather-forecasts --location 33.4366,126.5269 --store-in-db 
 
-.. note::  DarkSky is not handing out tokens anymore, as they have been bought by Apple (see `issue 3 <https://github.com/SeitaBV/flexmeasures/issues/56>`_).
+.. note::  DarkSky is not handing out tokens any more, as they have been bought by Apple (see `issue 3 <https://github.com/SeitaBV/flexmeasures/issues/56>`_).
 
 
 Preparing the job queue database and start workers

diff --git a/documentation/index.rst b/documentation/index.rst
@@ -85,6 +85,7 @@ The platform operator of FlexMeasures can be an Aggregator.
     dev/api
     dev/ci
     dev/plugins
+    dev/modes
 
 .. toctree::
     :caption: Integrations

diff --git a/flexmeasures/api/common/responses.py b/flexmeasures/api/common/responses.py
@@ -64,7 +64,7 @@ def invalid_domain(message: str) -> ResponseTuple:
     return dict(result="Rejected", status="INVALID_DOMAIN", message=message), 400
 
 
-@BaseMessage("The prognosis horizon in your request could not be parsed.")
+@BaseMessage("The horizon field in your request could not be parsed.")
 def invalid_horizon(message: str) -> ResponseTuple:
     return dict(result="Rejected", status="INVALID_HORIZON", message=message), 400
 
@@ -86,7 +86,7 @@ def invalid_ptu_duration(message: str) -> ResponseTuple:
     )
 
 
-@BaseMessage("Only the following resolutions are supported:")
+@BaseMessage("Only the following resolutions in the data are supported:")
 def unapplicable_resolution(message: str) -> ResponseTuple:
     return dict(result="Rejected", status="INVALID_RESOLUTION", message=message), 400
 

diff --git a/flexmeasures/api/common/utils/validators.py b/flexmeasures/api/common/utils/validators.py
@@ -296,17 +296,25 @@ def decorated_service(*args, **kwargs):
     return wrapper
 
 
-def optional_prior_accepted(ex_post: bool = False, infer_missing: bool = True):
+def optional_prior_accepted(
+    ex_post: bool = False, infer_missing: bool = True, infer_missing_play: bool = False
+):
     """Decorator which specifies that a GET or POST request accepts an optional prior.
     It parses relevant form data and sets the "prior" keyword param.
 
     Interpretation for GET requests:
     -   Denotes "at least before <prior>"
     -   This results in the filter belief_time_window = (None, prior)
 
-    Optionally, an ex_post flag can be passed to the decorator to indicate that only ex-post datetimes are allowed.
-    As a useful setting (at least for POST requests), set infer_missing to True to have servers
-    (that are not in play mode) derive a prior from the server time.
+    Interpretation for POST requests:
+    -   Denotes "recorded <prior> to some datetime,
+    -   this results in the assignment belief_time = prior
+
+    :param ex_post:                     if True, only ex-post datetimes are allowed.
+    :param infer_missing:               if True, servers assume that the belief_time of posted
+                                        values is server time. This setting is meant to be used for POST requests.
+    :param infer_missing_play:          if True, servers in play mode assume that the belief_time of posted
+                                        values is server time. This setting is meant to be used for POST requests.
     """
 
     def wrapper(fn):
@@ -332,11 +340,11 @@ def decorated_service(*args, **kwargs):
                     if prior < knowledge_time:
                         extra_info = "Meter data can only be observed after the fact."
                         return invalid_horizon(extra_info)
-            elif (
-                infer_missing is True
-                and current_app.config.get("FLEXMEASURES_MODE", "") != "play"
+            elif infer_missing is True or (
+                infer_missing_play is True
+                and current_app.config.get("FLEXMEASURES_MODE", "") == "play"
             ):
-                # A missing prior is inferred by the server (if not in play mode)
+                # A missing prior is inferred by the server
                 prior = server_now()
             else:
                 # Otherwise, a missing prior is fine (a horizon may still be inferred by the server)
@@ -353,6 +361,7 @@ def decorated_service(*args, **kwargs):
 def optional_horizon_accepted(  # noqa C901
     ex_post: bool = False,
     infer_missing: bool = True,
+    infer_missing_play: bool = False,
     accept_repeating_interval: bool = False,
 ):
     """Decorator which specifies that a GET or POST request accepts an optional horizon.
@@ -376,11 +385,13 @@ def optional_horizon_accepted(  # noqa C901
         def post_meter_data(horizon):
             return 'Meter data posted'
 
-    :param ex_post:                   if True, only non-positive horizons are allowed.
-    :param infer_missing:             if True, servers that are in play mode assume that the belief_horizon of posted
-                                      values is 0 hours. This setting is meant to be used for POST requests.
-    :param accept_repeating_interval: if True, the "rolling" keyword param is also set
-                                      (this was used for POST requests before v2.0)
+    :param ex_post:                     if True, only non-positive horizons are allowed.
+    :param infer_missing:               if True, servers assume that the belief_horizon of posted
+                                        values is 0 hours. This setting is meant to be used for POST requests.
+    :param infer_missing_play:          if True, servers in play mode assume that the belief_horizon of posted
+                                        values is 0 hours. This setting is meant to be used for POST requests.
+    :param accept_repeating_interval:   if True, the "rolling" keyword param is also set
+                                        (this was used for POST requests before v2.0)
     """
 
     def wrapper(fn):
@@ -410,15 +421,12 @@ def decorated_service(*args, **kwargs):
                         "For example: R/P1D should be replaced by P1D."
                     )
                     return invalid_horizon(extra_info)
-            elif (
-                infer_missing is True
+            elif infer_missing is True or (
+                infer_missing_play is True
                 and current_app.config.get("FLEXMEASURES_MODE", "") == "play"
             ):
-                # A missing horizon is set to zero for servers in play mode
+                # A missing horizon is set to zero
                 horizon = timedelta(hours=0)
-            elif infer_missing is True and accept_repeating_interval is True:
-                extra_info = "Missing horizons are no longer accepted for API versions below v2.0."
-                return invalid_horizon(extra_info)
             else:
                 # Otherwise, a missing horizon is fine (a prior may still be inferred by the server)
                 horizon = None

diff --git a/flexmeasures/api/v1/implementations.py b/flexmeasures/api/v1/implementations.py
@@ -97,7 +97,9 @@ def get_meter_data_response(
 @units_accepted("power", "MW")
 @assets_required("connection")
 @values_required
-@optional_horizon_accepted(ex_post=True, accept_repeating_interval=True)
+@optional_horizon_accepted(
+    ex_post=True, infer_missing=True, accept_repeating_interval=True
+)
 @period_required
 @post_data_checked_for_required_resolution("connection")
 @as_json