small doc improvements from user testing

documentation/api/change_log.rst

@@ -3,12 +3,11 @@
 API change log
 ===============
 
-v2.0 | 2021-04-02
+v2.0-2 | 2021-04-02
 """""""""""""""""
 
 - [**Breaking change**] Switched the interpretation of horizons to rolling horizons.
 - [**Breaking change**] Deprecated the use of ISO 8601 repeating time intervals to denote rolling horizons.
-- [**Breaking change**] Deprecated the automatic inference of horizons for *postMeterData*, *postPrognosis*, *postPriceData* and *postWeatherData* endpoints for API version below v2.0.
 - Introduced the "prior" field for *postMeterData*, *postPrognosis*, *postPriceData* and *postWeatherData* endpoints.
 - Changed the Introduction section:
 
@@ -18,16 +17,31 @@ v2.0 | 2021-04-02
 
     - Rewrote relevant examples using horizon and prior fields.
 
-v2.0 | 2021-02-19
+v2.0-1 | 2021-02-19
 """""""""""""""""""
 
 - REST endpoints for managing users: `/users/` (GET), `/user/<id>` (GET, PATCH) and `/user/<id>/password-reset` (PATCH).
 
-v2.0 | 2020-11-14
+v2.0-0 | 2020-11-14
 """""""""""""""""""
 
 - REST endpoints for managing assets: `/assets/` (GET, POST) and `/asset/<id>` (GET, PATCH, DELETE).
 
+
+v1.3.9 | 2021-04-XX
+"""""""""""""""""
+
+*Affects all versions since v1.0*.
+
+- Fixed regression by partially reverting the breaking change of v1.3-8: Re-instantiated automatic inference of horizons for Post requests for API versions below v2.0, but changed to inference policy: now inferring the data was recorded **right after each event** took place (leading to a zero horizon for each data point) rather than **after the last event** took place (which led to a different horizon for each data point); the latter had been the inference policy before v1.3-8.
+
+v1.3-8 | 2020-04-02
+"""""""""""""""""""
+
+*Affects all versions since v1.0*.
+
+- [**Breaking change**, partially reverted in v1.3-9] Deprecated the automatic inference of horizons for *postMeterData*, *postPrognosis*, *postPriceData* and *postWeatherData* endpoints for API version below v2.0.
+
 v1.3-7 | 2020-12-16
 """""""""""""""""""
 

documentation/api/introduction.rst

@@ -257,7 +257,9 @@ In case of a single group of connections, the message may be flattened to:
 Timeseries
 ^^^^^^^^^^
 
-Timestamps and durations are consistent with the ISO 8601 standard. All timestamps in requests to the API must be timezone-aware. The timezone indication "Z" indicates a zero offset from UTC. Additionally, we use the following shorthand for sequential values within a time interval:
+Timestamps and durations are consistent with the ISO 8601 standard. The resolution of the data is implicit, see :ref:`resolutions`.
+
+All timestamps in requests to the API must be timezone-aware. The timezone indication "Z" indicates a zero offset from UTC. Additionally, we use the following shorthand for sequential values within a time interval:
 
 .. code-block:: json
 
@@ -405,8 +407,9 @@ This denotes that the prognosed interval has 5 minutes left to be concluded.
 Resolutions
 ^^^^^^^^^^^
 
-Specifying a resolution is redundant for POST requests that contain both "values" and a "duration".
-Also, posted data is checked against the required resolution of the assets which are posted to.
+Specifying a resolution is redundant for POST requests that contain both "values" and a "duration" ― FlexMeasures computes the resolution by dividing the duration by the number of values.
+
+When POSTing data, FlexMeasures checks this computed resolution against the required resolution of the assets which are posted to. If these can't be matched (through upsampling), an error will occur.
 
 GET requests (such as *getMeterData*) return data in the resolution which the sensor is configured for.
 A "resolution" may be specified explicitly to obtain the data in downsampled form, 

documentation/dev/data.rst

@@ -100,36 +100,35 @@ Or, from within Postgres console:
    CREATE DATABASE flexmeasures_test WITH OWNER = flexmeasures_test;
 
 
-Log in as the postgres superuser and connect to your newly-created database:
+Finally, test if you can log in as the flexmeasures user:
 
 .. code-block:: bash
 
-   sudo -u postgres psql
+   psql -U flexmeasures --password -h 127.0.0.1 -d flexmeasures
 
 .. code-block:: sql
 
-   \connect flexmeasures
+   \q
+
 
+Add Postgres Extensions to your database(s)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
+To find the nearest sensors, FlexMeasures needs some extra POstgres support. 
 Add the following extensions while logged in as the postgres superuser:
 
+.. code-block:: bash
+
+   sudo -u postgres psql
+
 .. code-block:: sql
 
+   \connect flexmeasures
    CREATE EXTENSION cube;
    CREATE EXTENSION earthdistance;
 
 
-Connect to the ``flexmeasures_test`` database and repeat creating these extensions there. Then ``exit``.
-
-Finally, try logging in as the flexmeasures user once:
-
-.. code-block:: bash
-
-   psql -U flexmeasures --password -h 127.0.0.1 -d flexmeasures
-
-.. code-block:: sql
-
-   \q
+If you have it, connect to the ``flexmeasures_test`` database and repeat creating these extensions there. Then ``exit``.
 
 
 Configure FlexMeasures app for that database

documentation/dev/plugins.rst

@@ -76,4 +76,21 @@ All else that is needed for this showcase (not shown here) is ``<some_folder>/ou
 
 
 
-.. note:: Plugin views can also be added to the FlexMeasures UI menu ― just name them in the config setting :ref:`menu-config`.
+.. note:: Plugin views can also be added to the FlexMeasures UI menu ― just name them in the config setting :ref:`menu-config`.
+
+
+Using other files in your plugin
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Say you want to include other Python files in your plugin, importing them in your ``__init__.py`` file.
+This can be done if you put the plugin path on the import path. Do it like this in your ``__init__.py``:
+
+.. code-block:: python
+
+    import os
+    import sys
+
+    HERE = os.path.dirname(os.path.abspath(__file__))
+    sys.path.insert(0, HERE)
+
+    from my_other_file import my_function

documentation/getting-started.rst

@@ -17,6 +17,7 @@ Install dependencies and the ``flexmeasures`` platform itself:
 
    pip install flexmeasures
 
+.. note:: With newer Python versions and Windows, some smaller dependencies (e.g. ``tables`` or ``rq-win``) might cause issues as support is often slower. You might overcome this with a little research, by `installing from wheels <http://www.pytables.org/usersguide/installation.html#prerequisitesbininst>`_ or `from the repo <https://github.com/michaelbrooks/rq-win#installation-and-use>`_, respectively.
 
 
 Make a secret key for sessions and password salts
@@ -217,7 +218,7 @@ To collect weather measurements and forecasts from the DarkSky API, there is a t
 
    flexmeasures add external-weather-forecasts --location 33.4366,126.5269 --store-in-db 
 
-.. note::  DarkSky is not handing out tokens anymore, as they have been bought by Apple (see `issue 3 <https://github.com/SeitaBV/flexmeasures/issues/56>`_).
+.. note::  DarkSky is not handing out tokens any more, as they have been bought by Apple (see `issue 3 <https://github.com/SeitaBV/flexmeasures/issues/56>`_).
 
 
 Preparing the job queue database and start workers

flexmeasures/api/common/responses.py

@@ -86,7 +86,7 @@ def invalid_ptu_duration(message: str) -> ResponseTuple:
     )
 
 
-@BaseMessage("Only the following resolutions are supported:")
+@BaseMessage("Only the following resolutions in the data are supported:")
 def unapplicable_resolution(message: str) -> ResponseTuple:
     return dict(result="Rejected", status="INVALID_RESOLUTION", message=message), 400
 

flexmeasures/api/common/utils/validators.py

@@ -306,7 +306,7 @@ def optional_prior_accepted(ex_post: bool = False, infer_missing: bool = True):
 
     Optionally, an ex_post flag can be passed to the decorator to indicate that only ex-post datetimes are allowed.
     As a useful setting (at least for POST requests), set infer_missing to True to have servers
-    (that are not in play mode) derive a prior from the server time.
+    derive a prior from the server time.
     """
 
     def wrapper(fn):
@@ -332,11 +332,8 @@ def decorated_service(*args, **kwargs):
                     if prior < knowledge_time:
                         extra_info = "Meter data can only be observed after the fact."
                         return invalid_horizon(extra_info)
-            elif (
-                infer_missing is True
-                and current_app.config.get("FLEXMEASURES_MODE", "") != "play"
-            ):
-                # A missing prior is inferred by the server (if not in play mode)
+            elif infer_missing is True:
+                # A missing prior is inferred by the server
                 prior = server_now()
             else:
                 # Otherwise, a missing prior is fine (a horizon may still be inferred by the server)
@@ -377,7 +374,7 @@ def post_meter_data(horizon):
             return 'Meter data posted'
 
     :param ex_post:                   if True, only non-positive horizons are allowed.
-    :param infer_missing:             if True, servers that are in play mode assume that the belief_horizon of posted
+    :param infer_missing:             if True, servers assume that the belief_horizon of posted
                                       values is 0 hours. This setting is meant to be used for POST requests.
     :param accept_repeating_interval: if True, the "rolling" keyword param is also set
                                       (this was used for POST requests before v2.0)
@@ -410,15 +407,9 @@ def decorated_service(*args, **kwargs):
                         "For example: R/P1D should be replaced by P1D."
                     )
                     return invalid_horizon(extra_info)
-            elif (
-                infer_missing is True
-                and current_app.config.get("FLEXMEASURES_MODE", "") == "play"
-            ):
-                # A missing horizon is set to zero for servers in play mode
+            elif infer_missing is True:
+                # A missing horizon is set to zero
                 horizon = timedelta(hours=0)
-            elif infer_missing is True and accept_repeating_interval is True:
-                extra_info = "Missing horizons are no longer accepted for API versions below v2.0."
-                return invalid_horizon(extra_info)
             else:
                 # Otherwise, a missing horizon is fine (a prior may still be inferred by the server)
                 horizon = None

flexmeasures/api/v2_0/implementations/sensors.py

@@ -46,8 +46,16 @@
 @type_accepted("PostPriceDataRequest")
 @units_accepted("price", "EUR/MWh", "KRW/kWh")
 @assets_required("market")
-@optional_horizon_accepted()
-@optional_prior_accepted()
+@optional_horizon_accepted(
+    infer_missing=True
+    if current_app.config.get("FLEXMEASURES_MODE", "") == "play"
+    else False
+)
+@optional_prior_accepted(
+    infer_missing=False
+    if current_app.config.get("FLEXMEASURES_MODE", "") == "play"
+    else True
+)
 @values_required
 @period_required
 @post_data_checked_for_required_resolution("market")
@@ -130,8 +138,16 @@ def post_price_data_response(  # noqa C901
 @type_accepted("PostWeatherDataRequest")
 @unit_required
 @assets_required("sensor")
-@optional_horizon_accepted()
-@optional_prior_accepted()
+@optional_horizon_accepted(
+    infer_missing=True
+    if current_app.config.get("FLEXMEASURES_MODE", "") == "play"
+    else False
+)
+@optional_prior_accepted(
+    infer_missing=False
+    if current_app.config.get("FLEXMEASURES_MODE", "") == "play"
+    else True
+)
 @values_required
 @period_required
 @post_data_checked_for_required_resolution("sensor")
@@ -222,8 +238,18 @@ def post_weather_data_response(  # noqa: C901
 @units_accepted("power", "MW")
 @assets_required("connection")
 @values_required
-@optional_horizon_accepted(ex_post=True)
-@optional_prior_accepted(ex_post=True)
+@optional_horizon_accepted(
+    ex_post=True,
+    infer_missing=True
+    if current_app.config.get("FLEXMEASURES_MODE", "") == "play"
+    else False,
+)
+@optional_prior_accepted(
+    ex_post=True,
+    infer_missing=False
+    if current_app.config.get("FLEXMEASURES_MODE", "") == "play"
+    else True,
+)
 @period_required
 @post_data_checked_for_required_resolution("connection")
 @as_json
@@ -254,8 +280,13 @@ def post_meter_data_response(
 @units_accepted("power", "MW")
 @assets_required("connection")
 @values_required
-@optional_horizon_accepted(ex_post=False)
-@optional_prior_accepted(ex_post=False)
+@optional_horizon_accepted(ex_post=False, infer_missing=False)
+@optional_prior_accepted(
+    ex_post=False,
+    infer_missing=False
+    if current_app.config.get("FLEXMEASURES_MODE", "") == "play"
+    else True,
+)
 @period_required
 @post_data_checked_for_required_resolution("connection")
 @as_json

flexmeasures/data/scripts/cli_tasks/data_add.py

@@ -230,14 +230,14 @@ def add_initial_structure():
     multiple=True,
     type=click.Choice(["1", "6", "24", "48"]),
     default=["1", "6", "24", "48"],
-    help="Forecasting horizon in hours. This argument can be given multiple times.",
+    help="Forecasting horizon in hours. This argument can be given multiple times. Defaults to all possible horizons.",
 )
 @click.option(
     "--as-job",
     is_flag=True,
     help="Whether to queue a forecasting job instead of computing directly."
     " Useful to run locally and create forecasts on a remote server. In that case, just point the redis db in your"
-    " config settings to that of the remote server. To process the job, run a worker to process the forecasting queue.",
+    " config settings to that of the remote server. To process the job, run a worker to process the forecasting queue. Defaults to False.",
 )
 def create_forecasts(
     asset_type: str = None,

to_pypi.sh

@@ -8,24 +8,24 @@
 # The version comes from setuptools_scm. See `python setup.py --version`.
 # setuptools_scm works via git tags that should implement a semantic versioning scheme, e.g. v0.2.3
 #
-# If there were zero commits since since tag, we have a real release and the version basicaly *is* what the tag says.
-# Otherwise, the version also include a .devN identifier, where N is the number of commits since the last version tag.
+# If there were zero commits since since tag, we have a real release and the version basically *is* what the tag says.
+# Otherwise, the version also includes a .devN identifier, where N is the number of commits since the last version tag.
 #
 # More information on creating a dev release
 # -------------------------------------------
 # Note that the only way to create a new dev release is to add another commit on your development branch.
-# It might have been convenient to not have to commit to do that (for exoerimenting with very small changes),
+# It might have been convenient to not have to commit to do that (for experimenting with very small changes),
 # but we decided against that. Let's explore why for a bit:
 #
 # First, setuptools_scm has the ability to add a local scheme (git commit and date/time) to the version,
 # but we've disabled that, as that extra part isn't formatted in a way that Pypi accepts it.
-# Another way would have been to add a local version identifier ("+M", not the plus sign),
+# Another way would have been to add a local version identifier ("+M", note the plus sign),
 # which is allowed in PEP 440 but explicitly disallowed by Pypi.
 # Finally, if we simply add a number to .devN (-> .devNM), the ordering of dev versions would be
 # disturbed after the next local commit (e.g. we add 1 to .dev4, making it .dev41, and then the next version, .dev5,
 # is not the highest version chosen by PyPi).
 # 
-# So we'll use these tools as the experts intend us to.
+# So we'll use these tools as the experts intended.
 # If you want, you can read more about acceptable versions in PEP 440: https://www.python.org/dev/peps/pep-0440/
 
 
@@ -35,4 +35,4 @@ pip -q install twine
 python setup.py egg_info sdist
 python setup.py egg_info bdist_wheel
 
-twine upload dist/*
+twine upload dist/*