merge master and get tests to work

documentation/api/change_log.rst

@@ -3,12 +3,14 @@
 API change log
 ===============
 
-v2.0 | 2021-04-02
+.. note:: The FlexMeasures API follows its own versioning scheme. This also reflects in the URL, allowing developers to upgrade at their own pace.
+
+
+v2.0-2 | 2021-04-02
 """""""""""""""""
 
 - [**Breaking change**] Switched the interpretation of horizons to rolling horizons.
 - [**Breaking change**] Deprecated the use of ISO 8601 repeating time intervals to denote rolling horizons.
-- [**Breaking change**] Deprecated the automatic inference of horizons for *postMeterData*, *postPrognosis*, *postPriceData* and *postWeatherData* endpoints for API version below v2.0.
 - Introduced the "prior" field for *postMeterData*, *postPrognosis*, *postPriceData* and *postWeatherData* endpoints.
 - Changed the Introduction section:
 
@@ -18,16 +20,31 @@ v2.0 | 2021-04-02
 
     - Rewrote relevant examples using horizon and prior fields.
 
-v2.0 | 2021-02-19
+v2.0-1 | 2021-02-19
 """""""""""""""""""
 
 - REST endpoints for managing users: `/users/` (GET), `/user/<id>` (GET, PATCH) and `/user/<id>/password-reset` (PATCH).
 
-v2.0 | 2020-11-14
+v2.0-0 | 2020-11-14
 """""""""""""""""""
 
 - REST endpoints for managing assets: `/assets/` (GET, POST) and `/asset/<id>` (GET, PATCH, DELETE).
 
+
+v1.3.9 | 2021-04-XX
+"""""""""""""""""
+
+*Affects all versions since v1.0*.
+
+- Fixed regression by partially reverting the breaking change of v1.3-8: Re-instantiated automatic inference of horizons for Post requests for API versions below v2.0, but changed to inference policy: now inferring the data was recorded **right after each event** took place (leading to a zero horizon for each data point) rather than **after the last event** took place (which led to a different horizon for each data point); the latter had been the inference policy before v1.3-8.
+
+v1.3-8 | 2020-04-02
+"""""""""""""""""""
+
+*Affects all versions since v1.0*.
+
+- [**Breaking change**, partially reverted in v1.3-9] Deprecated the automatic inference of horizons for *postMeterData*, *postPrognosis*, *postPriceData* and *postWeatherData* endpoints for API version below v2.0.
+
 v1.3-7 | 2020-12-16
 """""""""""""""""""
 
@@ -155,10 +172,7 @@ v1.1-2 | 2018-08-15
 - Added the *postPriceData* endpoint
 - Added a description of the *postPrognosis* endpoint in the Aggregator section
 - Added a description of the *postPriceData* endpoint in the Aggregator and Supplier sections
-
-.. ifconfig:: FLEXMEASURES_MODE == "play"
-
-    - Added the *restoreData* endpoint
+- Added the *restoreData* endpoint for servers in play mode
 
 v1.1-1 | 2018-08-06
 """""""""""""""""""

documentation/api/introduction.rst

@@ -283,7 +283,9 @@ In case of a single group of connections, the message may be flattened to:
 Timeseries
 ^^^^^^^^^^
 
-Timestamps and durations are consistent with the ISO 8601 standard. All timestamps in requests to the API must be timezone-aware. The timezone indication "Z" indicates a zero offset from UTC. Additionally, we use the following shorthand for sequential values within a time interval:
+Timestamps and durations are consistent with the ISO 8601 standard. The resolution of the data is implicit, see :ref:`resolutions`.
+
+All timestamps in requests to the API must be timezone-aware. The timezone indication "Z" indicates a zero offset from UTC. Additionally, we use the following shorthand for sequential values within a time interval:
 
 .. code-block:: json
 
@@ -431,8 +433,9 @@ This denotes that the prognosed interval has 5 minutes left to be concluded.
 Resolutions
 ^^^^^^^^^^^
 
-Specifying a resolution is redundant for POST requests that contain both "values" and a "duration".
-Also, posted data is checked against the required resolution of the assets which are posted to.
+Specifying a resolution is redundant for POST requests that contain both "values" and a "duration" ― FlexMeasures computes the resolution by dividing the duration by the number of values.
+
+When POSTing data, FlexMeasures checks this computed resolution against the required resolution of the assets which are posted to. If these can't be matched (through upsampling), an error will occur.
 
 GET requests (such as *getMeterData*) return data in the resolution which the sensor is configured for.
 A "resolution" may be specified explicitly to obtain the data in downsampled form, 

documentation/changelog.rst

@@ -3,17 +3,70 @@ FlexMeasures Changelog
 **********************
 
 
-v0.2.5 | April XX, 2021
+v0.5.0 | May XX, 2021
 ===========================
 
+.. warning:: If you retrieve weather forecasts through FlexMeasures: we had to switch to OpenWeatherMap, as Dark Sky is closing. This requires an update to config variables ― the new setting is called ``OPENWEATHERMAP_API_KEY``.
+
 New features
 -----------
-* Add sensors with CLI command [see `PR #83 <https://github.com/SeitaBV/flexmeasures/pull/83>`_]
+* Allow plugins to overwrite UI routes and customise the teaser on the login form [see `PR #106 <http://www.github.com/SeitaBV/flexmeasures/pull/106>`_]
+* Allow plugins to customise the copyright notice and credits in the UI footer [see `PR #123 <http://www.github.com/SeitaBV/flexmeasures/pull/123>`_]
+
+Bugfixes
+-----------
+* Fix last login date display in user list [see `PR #133 <http://www.github.com/SeitaBV/flexmeasures/pull/133>`_]
+* Choose better forecasting horizons when weather data is posted [see `PR #131 <http://www.github.com/SeitaBV/flexmeasures/pull/131>`_]
+
+Infrastructure / Support
+----------------------
+* Make assets use MW as their default unit and enforce that in CLI, as well (API already did) [see `PR #108 <http://www.github.com/SeitaBV/flexmeasures/pull/108>`_]
+* For weather forecasts, switch from Dark Sky (closed from Aug 1, 2021) to OpenWeatherMap API [see `PR #113 <http://www.github.com/SeitaBV/flexmeasures/pull/113>`_]
+* Re-use the database between automated tests, if possible. This shaves 2/3rd off of the time it takes for the FlexMeasures test suite to run [see `PR #115 <http://www.github.com/SeitaBV/flexmeasures/pull/115>`_]
+* Let CLI package and plugins use Marshmallow Field definitions [see `PR #125 <http://www.github.com/SeitaBV/flexmeasures/pull/125>`_]
+
+
+v0.4.1 | May 7, 2021
+===========================
+
+Bugfixes
+-----------
+* Fix regression when editing assets in the UI [see `PR #122 <http://www.github.com/SeitaBV/flexmeasures/pull/122>`_]
+* Fixed a regression that stopped asset, market and sensor selection from working [see `PR #117 <http://www.github.com/SeitaBV/flexmeasures/pull/117>`_]
+* Prevent logging out user when clearing the session [see `PR #112 <http://www.github.com/SeitaBV/flexmeasures/pull/112>`_]
+* Prevent user type data source to be created without setting a user [see `PR #111 <https://github.com/SeitaBV/flexmeasures/pull/111>`_]
+
+v0.4.0 | April 29, 2021
+===========================
+
+.. warning:: Upgrading to this version requires running ``flexmeasures db upgrade`` (you can create a backup first with ``flexmeasures db-ops dump``).
+
+New features
+-----------
+* Configure the UI menu with ``FLEXMEASURES_LISTED_VIEWS`` [see `PR #91 <https://github.com/SeitaBV/flexmeasures/pull/91>`_]
+* Allow for views and CLI functions to come from plugins [see also `PR #91 <https://github.com/SeitaBV/flexmeasures/pull/91>`_]
+
+.. note:: Read more on these features on `the FlexMeasures blog <https://flexmeasures.io/v040-plugin-support/>`__.
+
+Bugfixes
+-----------
+* Asset edit form displayed wrong error message. Also enabled the asset edit form to display the invalid user input back to the user [see `PR #93 <http://www.github.com/SeitaBV/flexmeasures/pull/93>`_]
 
 Infrastructure / Support
 ----------------------
 * Updated dependencies, including Flask-Security-Too [see `PR #82 <http://www.github.com/SeitaBV/flexmeasures/pull/82>`_]
-* Integration with `timely beliefs <https://github.com/SeitaBV/timely-beliefs>`_ lib: Sensor data as TimedBeliefs [see `PR #79 <http://www.github.com/SeitaBV/flexmeasures/pull/79>`_]
+* Improved documentation after user feedback [see `PR #97 <http://www.github.com/SeitaBV/flexmeasures/pull/97>`_]
+* Begin experimental integration with `timely beliefs <https://github.com/SeitaBV/timely-beliefs>`_ lib: Sensor data as TimedBeliefs [see `PR #79 <http://www.github.com/SeitaBV/flexmeasures/pull/79>`_ and `PR #99 <https://github.com/SeitaBV/flexmeasures/pull/99>`_]
+* Add sensors with CLI command currently meant for developers only [see `PR #83 <https://github.com/SeitaBV/flexmeasures/pull/83>`_]
+* Add data (beliefs about sensor events) with CLI command currently meant for developers only [see `PR #85 <https://github.com/SeitaBV/flexmeasures/pull/85>`_ and `PR #103 <https://github.com/SeitaBV/flexmeasures/pull/103>`_]
+
+
+v0.3.1 | April 9, 2021
+===========================
+
+Bugfixes
+--------
+* PostMeterData endpoint was broken in API v2.0 [see `PR #95 <http://www.github.com/SeitaBV/flexmeasures/pull/95>`_]
 
 
 v0.3.0 | April 2, 2021

documentation/cli/change_log.rst

@@ -4,6 +4,11 @@
 FlexMeasures CLI Changelog
 **********************
 
+since v0.4.0 | April 2, 2021
+=====================
+
+* Add the ``dev-add`` command group for experimental features around the upcoming data model refactoring.
+
 
 since v0.3.0 | April 2, 2021
 =====================

documentation/concepts/services.rst

@@ -15,6 +15,8 @@ The FlexMeasures platform continuously reads in meter data from your assets. To
 * Data gaps & strange outliers (assure data quality)
 * Idle processes / leaks (minimise waste)
 
+.. todo:: These features are work in progress. Most of our customers already do this by themselves in a straightforward manner.
+
 
 Forecasting
 --------------

documentation/configuration.rst

@@ -6,7 +6,7 @@ Configuration
 The following configurations are used by FlexMeasures.
 
 Required settings (e.g. postgres db) are marked with a double star (**).
-To enable easier quickstart tutorials, these settings can be set by env vars.
+To enable easier quickstart tutorials, these settings can be set by environment variables.
 Recommended settings (e.g. mail, redis) are marked by one star (*).
 
 .. note:: FlexMeasures is best configured via a config file. The config file for FlexMeasures can be placed in one of two locations: 
@@ -15,6 +15,7 @@ Recommended settings (e.g. mail, redis) are marked by one star (*).
 * in the user's home directory (e.g. ``~/.flexmeasures.cfg`` on Unix). In this case, note the dot at the beginning of the filename!
 * in the app's instance directory (e.g. ``/path/to/your/flexmeasures/code/instance/flexmeasures.cfg``\ ). The path to that instance directory is shown to you by running flexmeasures (e.g. ``flexmeasures run``\ ) with required settings missing or otherwise by running ``flexmeasures shell``.
 
+
 Basic functionality
 -------------------
 
@@ -25,11 +26,14 @@ Level above which log messages are added to the log file. See the ``logging`` pa
 
 Default: ``logging.WARNING``
 
+
+.. _modes-config:
+
 FLEXMEASURES_MODE
 ^^^^^^^^^^^^^^^^^
 
 The mode in which FlexMeasures is being run, e.g. "demo" or "play".
-This is used to turn on certain extra behaviours.
+This is used to turn on certain extra behaviours, see :ref:`modes-dev` for details.
 
 Default: ``""``
 
@@ -51,6 +55,19 @@ and the first month when the domain was under the current owner's administration
 
 Default: ``{"flexmeasures.io": "2021-01"}``
 
+
+.. _plugin-config:
+
+FLEXMEASURES_PLUGIN_PATHS
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A list of absolute paths to Blueprint-based plugins for FlexMeasures (e.g. for custom views or CLI functions).
+Each plugin path points to a folder, which should contain an ``__init__.py`` file where the Blueprint is defined. 
+See :ref:`plugins` on what is expected for content.
+
+Default: ``[]``
+
+
 FLEXMEASURES_DB_BACKUP_PATH
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -65,6 +82,7 @@ Whether to turn on a feature which times requests made through FlexMeasures. Int
 
 Default: ``False``
 
+
 UI
 --
 
@@ -89,6 +107,7 @@ Interval in which viewing the queues dashboard refreshes itself, in milliseconds
 
 Default: ``3000`` (3 seconds) 
 
+
 Timing
 ------
 
@@ -113,18 +132,14 @@ The horizon to use when making schedules.
 
 Default: ``timedelta(hours=2 * 24)``
 
+
 Tokens
 ------
 
-DARK_SKY_API_KEY
+OPENWEATHERMAP_API_KEY
 ^^^^^^^^^^^^^^^^
 
-Token for accessing the DarkSky weather forecasting service.
-
-.. note:: DarkSky will soon become non-public (Aug 1, 2021), so they are not giving out new tokens.
-          We'll use another service soon (`see this issue <https://github.com/SeitaBV/flexmeasures/issues/3>`_).
-          This is unfortunate.
-          In the meantime, if you can't find anybody lending their token, consider posting weather forecasts to the FlexMeasures database yourself.
+Token for accessing the OPenWeatherMap weather forecasting service.
 
 Default: ``None``
 
@@ -133,7 +148,7 @@ Default: ``None``
 MAPBOX_ACCESS_TOKEN
 ^^^^^^^^^^^^^^^^^^^
 
-Token for accessing the mapbox API (for displaying maps on the dashboard and asset pages). You can learn how to obtain one `here <https://docs.mapbox.com/help/glossary/access-token/>`_
+Token for accessing the MapBox API (for displaying maps on the dashboard and asset pages). You can learn how to obtain one `here <https://docs.mapbox.com/help/glossary/access-token/>`_
 
 Default: ``None``
 
@@ -144,6 +159,7 @@ Token which external services can use to check on the status of recurring tasks
 
 Default: ``None``
 
+
 SQLAlchemy
 ----------
 
@@ -172,6 +188,7 @@ Default:
            "connect_args": {"options": "-c timezone=utc"},
        }
 
+
 Security
 --------
 
@@ -215,22 +232,22 @@ Default: ``60 * 60 * 6``  (six hours)
 SECURITY_TRACKABLE
 ^^^^^^^^^^^^^^^^^^
 
-Wether to track user statistics. Turning this on requires certain user fields.
+Whether to track user statistics. Turning this on requires certain user fields.
 We do not use this feature, but we do track number of logins.
 
 Default: ``False``
 
 CORS_ORIGINS
 ^^^^^^^^^^^^
 
-Allowed cross-origins. Set to "*" to allow all. For development (e.g. javascript on localhost) you might use "null" in this list.
+Allowed cross-origins. Set to "*" to allow all. For development (e.g. JavaScript on localhost) you might use "null" in this list.
 
 Default: ``[]``
 
 CORS_RESOURCES:
 ^^^^^^^^^^^^^^^
 
-FlexMeasures resources which get cors protection. This can be a regex, a list of them or dict with all possible options.
+FlexMeasures resources which get cors protection. This can be a regex, a list of them or a dictionary with all possible options.
 
 Default: ``[r"/api/*"]``
 
@@ -244,6 +261,7 @@ Allows users to make authenticated requests. If true, injects the Access-Control
 Default: ``True``
 
 
+
 .. _mail-config:
 
 Mail
@@ -335,7 +353,7 @@ Default: ``6379``
 FLEXMEASURES_REDIS_DB_NR (*)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Number of the redis database to use (Redis per default has 16 databases, nubered 0-15)
+Number of the redis database to use (Redis per default has 16 databases, numbered 0-15)
 
 Default: ``0``
 
@@ -349,13 +367,17 @@ Default: ``None``
 Demonstrations
 --------------
 
+.. _demo-credentials-config:
+
 FLEXMEASURES_PUBLIC_DEMO_CREDENTIALS
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 When ``FLEXMEASURES_MODE=demo``\ , this can hold login credentials (demo user email and password, e.g. ``("demo at seita.nl", "flexdemo")``\ ), so anyone can log in and try out the platform.
 
 Default: ``None``
 
+.. _demo-year-config:
+
 FLEXMEASURES_DEMO_YEAR
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -364,9 +386,14 @@ so that old imported data can be demoed as if it were current
 
 Default: ``None``
 
-FLEXMEASURES_SHOW_CONTROL_UI
+
+.. _menu-config:
+
+FLEXMEASURES_LISTED_VIEWS
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-The control page is still mocked, so this setting controls if it is to be shown.
+A list of the views which are listed in the menu.
 
-Default: ``False``
+.. note:: This setting is likely to be deprecated soon, as we might want to control it per account (once we implemented a multi-tenant data model per FlexMeasures server).
+
+Default: ``["dashboard", "analytics", "portfolio", "assets", "users"]``