First API tutorials (#130)

* add empty access to some 2_0 endpoints - this was breaking the getServices endpoint

* improve getService and getAuthToken part of API introduction

* move simulation part to new tutorial folder and rewrite the beginning

* first version of posting-data tutorial, prepare two others: forecasting/scheduling and custom-UIs

* finish forecasting&scheduling tutorial, finding an acceptable trade-off for explaining postUdiEvents; various smaller improvements

* tutorial about building custom UIs

* use company.flexmeasures.io as example URL in API introduction throughout

Co-authored-by: nhoening <nhoening@users.noreply.github.com>
Co-authored-by: Nicolas Höning <nicolas@seita.nl>
Co-authored-by: Nicolas Höning <iam@nicolashoening.de>
Co-authored-by: F.N. Claessen <felix@seita.nl>

Makefile

@@ -15,7 +15,7 @@ test:
 # ---- Documentation ---
 
 update-docs:
-	pip3 install sphinx sphinx-rtd-theme sphinxcontrib.httpdomain
+	pip3 install sphinx==3.5.4 sphinxcontrib.httpdomain # sphinx4 is not supported yet by sphinx-contrib/httpdomain, see https://github.com/sphinx-contrib/httpdomain/issues/46
 	cd documentation; make clean; make html; cd ..
 
 update-docs-pdf:

documentation/api/introduction.rst

@@ -4,28 +4,67 @@ Introduction
 ============
 
 This document details the Application Programming Interface (API) of the FlexMeasures web service. The API supports user automation for flexibility valorisation in the energy sector, both in a live setting and for the purpose of simulating scenarios. The web service adheres to the concepts and terminology used in the Universal Smart Energy Framework (USEF).
-We assume in this document that the FlexMeasures instance you want to connect to is hosted at https://company.flexmeasures.io.
 
 
-New versions of the API are released on:
+.. _api_versions:
+
+Main endpoint and API versions
+------------------------------
+
+All versions of the API are released on:
+
+.. code-block:: html
+
+    https://<flexmeasures-root-url>/api
+
+So if you are running FlexMeasures on your computer, it would be:
+
+.. code-block:: html
+
+    https://localhost:5000/api
+
+At Seita, we run servers for our clients at:
 
 .. code-block:: html
 
     https://company.flexmeasures.io/api
 
-A list of services offered by (a version of) the FlexMeasures web service can be obtained by sending a *getService* request. An optional field "access" can be used to specify a user role for which to obtain only the relevant services.
+where `company` is a hosting customer of ours. All their accounts' data lives on that server.
 
-**Example request**
+We assume in this document that the FlexMeasures instance you want to connect to is hosted at https://company.flexmeasures.io.
 
-.. code-block:: json
+Let's see what the ``/api`` endpoint returns:
 
-    {
-        "type": "GetServiceRequest",
-        "version": "1.0"
+.. code-block:: python
+
+    >>> import requests
+    >>> res = requests.get("https://company.flexmeasures.io/api")
+    >>> res.json()
+    {'flexmeasures_version': '0.4.0',
+     'message': 'For these API versions a public endpoint is available, listing its service. For example: /api/v1/getService and /api/v1_1/getService. An authentication token can be requested at: /api/requestAuthToken',
+     'status': 200,
+     'versions': ['v1', 'v1_1', 'v1_2', 'v1_3', 'v2_0']
     }
 
-**Example response**
+So this tells us which API versions exist. For instance, we know that the latest API version is available at
+
+.. code-block:: html
+
+    https://company.flexmeasures.io/api/v2_0
+
+
+Also, we can see that a list of endpoints which are available at (a version of) the FlexMeasures web service can be obtained by sending a ``getService`` request. An optional field "access" can be used to specify a user role for which to obtain only the relevant services.
+
+**Example request**
 
+Let's ask which endpoints are available for meter data companies (MDC):
+
+.. code-block:: html
+
+    https://company.flexmeasures.io/api/v2_0/getService?access=MDC
+
+
+**Example response**
 
 .. code-block:: json
 
@@ -46,6 +85,8 @@ A list of services offered by (a version of) the FlexMeasures web service can be
         ]
     }
 
+.. _api_auth:
+
 Authentication
 --------------
 
@@ -78,6 +119,15 @@ using the following JSON message for the POST request data:
         "password": "<user password>"
     }
 
+which gives a response like this if the credentials are correct:
+
+.. code-block:: json
+
+    {
+        "auth_token": "<authentication token>",
+        "user_id": "<ID of the user>"
+    }
+
 .. note:: Each access token has a limited lifetime, see :ref:`auth`.
 
 
@@ -257,9 +307,11 @@ In case of a single group of connections, the message may be flattened to:
 Timeseries
 ^^^^^^^^^^
 
-Timestamps and durations are consistent with the ISO 8601 standard. The resolution of the data is implicit, see :ref:`resolutions`.
+Timestamps and durations are consistent with the ISO 8601 standard. The resolution of the data is implicit (from duration and number of values), see :ref:`resolutions`.
+
+All timestamps in requests to the API must be timezone-aware. For instance, in the below example, the timezone indication "Z" indicates a zero offset from UTC.
 
-All timestamps in requests to the API must be timezone-aware. The timezone indication "Z" indicates a zero offset from UTC. Additionally, we use the following shorthand for sequential values within a time interval:
+We use the following shorthand for sending sequential, equidistant values within a time interval:
 
 .. code-block:: json
 
@@ -273,7 +325,7 @@ All timestamps in requests to the API must be timezone-aware. The timezone indic
         "duration": "PT45M"
     }
 
-is equal to:
+Technically, this is equal to:
 
 .. code-block:: json
 
@@ -302,31 +354,40 @@ This intuitive convention allows us to reduce communication by sending univariat
 Notation for v1
 """""""""""""""
 
-For version 1 of the API, only univariate timeseries data is expected to be communicated. Therefore:
+For version 1 and 2 of the API, only equidistant timeseries data is expected to be communicated. Therefore:
+
+- only the array notation should be used (first notation from above),
+- "start" should be a timestamp on the hour or a multiple of the sensor resolution thereafter (e.g. "16:10" works if the resolution is 5 minutes), and
+- "duration" should also be a multiple of the sensor resolution.
 
-- only the array notation should be used,
-- "start" should be a timestamp on the hour or a multiple of 15 minutes thereafter, and
-- "duration" should be a multiple of 15 minutes.
 
 .. _beliefs:
 
-Beliefs
-^^^^^^^
+Tracking the recording time of beliefs
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+For all its time series data, FlexMeasures keeps track of the time they were recorded. Data can be defined and filtered accordingly, which allows you to get a snapshot of what was known at a certain point in time.
+
+.. note:: FlexMeasures uses the `timely-beliefs data model <https://github.com/SeitaBV/timely-beliefs/#the-data-model>`_ for modelling such facts about time series data, and accordingly we use the term "belief" in this documentation. In that model, the recording time is referred to as "belief time".
+
+
+Querying by recording time
+""""""""""""""""""""""""""""
 
-By regarding all time series data as beliefs that have been recorded at a certain time, data can be filtered accordingly.
 Some GET endpoints have two optional timing fields to allow such filtering.
-The "prior" field (a timestamp) can be used to select beliefs recorded before some moment in time.
+
+The ``prior`` field (a timestamp) can be used to select beliefs recorded before some moment in time.
 It can be used to "time-travel" to see the state of information at some moment in the past.
-In addition, the "horizon" field (a duration) can be used to select beliefs recorded before some moment in time, relative to each event.
+
+In addition, the ``horizon`` field (a duration) can be used to select beliefs recorded before some moment in time, `relative to each event`.
 For example, to filter out meter readings communicated within a day (denoted by a negative horizon) or forecasts created at least a day beforehand (denoted by a positive horizon).
-In addition to these two timing filters, beliefs can be filtered by their source (see :ref:`sources`).
 
 The two timing fields follow the ISO 8601 standard and are interpreted as follows:
 
-- "horizon": recorded at least <duration> before the fact (indicated by a positive horizon), or at most <duration> after the fact (indicated by a negative horizon).
-- "prior": recorded prior to <timestamp>.
+- ``prior``: recorded prior to <timestamp>.
+- ``horizon``: recorded at least <duration> before the fact (indicated by a positive horizon), or at most <duration> after the fact (indicated by a negative horizon).
 
-For example:
+For example (note that you can use both fields together):
 
 .. code-block:: json
 
@@ -337,18 +398,21 @@ For example:
 
 These fields denote that the data should have been recorded at least 6 hours before the fact (i.e. forecasts) and prior to 5 PM on August 1st 2020 (UTC).
 
+.. note:: In addition to these two timing filters, beliefs can be filtered by their source (see :ref:`sources`).
+
+
 .. _prognoses:
 
-Prognoses
-^^^^^^^^^
+Setting the recording time
+""""""""""""""""""""""""
 
-Some POST endpoints have two optional fields to allow setting the time at which beliefs are recorded explicitly.
+Some POST endpoints have two optional fields to allow setting the time at which beliefs are recorded in an explicit manner.
 This is useful to keep an accurate history of what was known at what time, especially for prognoses.
-If not used, FlexMeasures will infer the prior from the arrival time of the message.
+If not used, FlexMeasures will infer the belief time from the arrival time of the message.
 
-The "prior" field (a timestamp) can be used to set a single time at which the entire prognosis was recorded.
-Alternatively, the "horizon" field (a duration) can be used to set the recording times relative to each prognosed event.
-In case both fields are set, the earliest possible recording time is determined and recorded for each prognosed event.
+The "prior" field (a timestamp) can be used to set a single time at which the entire time series (e.g. a prognosed series) was recorded.
+Alternatively, the "horizon" field (a duration) can be used to set the recording times relative to each (prognosed) event.
+In case both fields are set, the earliest possible recording time is determined and recorded for each (prognosed) event.
 
 The two timing fields follow the ISO 8601 standard and are interpreted as follows:
 
@@ -383,7 +447,7 @@ This message implies that the entire prognosis was recorded at 7:45 AM UTC, i.e.
 This message implies that all prognosed values were recorded 6 hours in advance.
 That is, the value for 1:00-1:15 PM was made at 7:15 AM, the value for 1:15-1:30 PM was made at 7:30 AM, and the value for 1:30-1:45 PM was made at 7:45 AM.
 
-Negative horizons may also be stated (breaking with the ISO 8601 standard) to indicate a prognosis about something that has already happened (i.e. after the fact, or simply *ex post*).
+Negative horizons may also be stated (breaking with the ISO 8601 standard) to indicate a belief about something that has already happened (i.e. after the fact, or simply *ex post*).
 For example, the following message implies that all prognosed values were made 10 minutes after the fact:
 
 .. code-block:: json
@@ -399,7 +463,7 @@ For example, the following message implies that all prognosed values were made 1
         "horizon": "-PT10M"
     }
 
-Note that, for a horizon indicating a prognosis 10 minutes after the *start* of each 15-minute interval, the "horizon" would have been "PT5M".
+Note that, for a horizon indicating a belief 10 minutes after the *start* of each 15-minute interval, the "horizon" would have been "PT5M".
 This denotes that the prognosed interval has 5 minutes left to be concluded.
 
 .. _resolutions:
@@ -425,8 +489,8 @@ Valid units for timeseries data in version 1 of the API are "MW" only.
 
 .. _signs:
 
-Signs
-^^^^^
+Signs of power values
+^^^^^^^^^^^^^^^^^^^^^
 
 USEF recommends to use positive power values to indicate consumption and negative values to indicate production, i.e.
 to take the perspective of the Prosumer.

documentation/changelog.rst

@@ -20,9 +20,10 @@ Bugfixes
 
 Infrastructure / Support
 ----------------------
-* Make assets use MW as their default unit and enforce that in CLI, as well (API already did) [see `PR #108 <http://www.github.com/SeitaBV/flexmeasures/pull/108>`_]
+* Add tutorials on how to add and read data from FlexMeasures via its API [see `PR #130 <http://www.github.com/SeitaBV/flexmeasures/pull/130>`_]
 * For weather forecasts, switch from Dark Sky (closed from Aug 1, 2021) to OpenWeatherMap API [see `PR #113 <http://www.github.com/SeitaBV/flexmeasures/pull/113>`_]
 * Re-use the database between automated tests, if possible. This shaves 2/3rd off of the time it takes for the FlexMeasures test suite to run [see `PR #115 <http://www.github.com/SeitaBV/flexmeasures/pull/115>`_]
+* Make assets use MW as their default unit and enforce that in CLI, as well (API already did) [see `PR #108 <http://www.github.com/SeitaBV/flexmeasures/pull/108>`_]
 * Let CLI package and plugins use Marshmallow Field definitions [see `PR #125 <http://www.github.com/SeitaBV/flexmeasures/pull/125>`_]
 * add time_utils.get_recent_clock_time_window() function [see `PR #135 <http://www.github.com/SeitaBV/flexmeasures/pull/135>`_]
 

documentation/concepts/algorithms.rst

@@ -9,6 +9,8 @@ Algorithms
     :depth: 2
 
 
+.. _algorithms_forecasting:
+
 Forecasting
 -----------
 
@@ -73,6 +75,8 @@ Improvements:
 - Most assets have yearly seasonality (e.g. wind, solar) and therefore forecasts would benefit from >= 2 years of history.
 
 
+.. _algorithms_scheduling:
+
 Scheduling 
 ------------
 

documentation/getting-started.rst

@@ -8,6 +8,9 @@ Quickstart
 
 This section walks you through getting FlexMeasures to run with the least effort. We'll cover making a secret key, connecting a database and creating one user & one asset.
 
+.. note:: Are you not hosting FlexMeasures, but want to learn how to use it? Head over to our tutorials, starting with :ref:`tut_posting_data`.
+
+
 Install FlexMeasures
 ^^^^^^^^^^^^^^^^^^^^
 
@@ -134,7 +137,7 @@ Add your first asset
 
 There are three ways to add assets:
 
-Head over to ``http://localhost:5000/assets`` and add a new asset there.
+Head over to ``http://localhost:5000/assets`` (after you started FlexMeasures, see next step) and add a new asset there in a web form.
 
 Or, use the ``flexmeasures`` :ref:`cli`:
 
@@ -209,39 +212,7 @@ We provide a script for installing from source (without requiring ``sudo`` right
 
 More information (e.g. for installing on Windows) on `the Cbc website <https://projects.coin-or.org/Cbc>`_.
 
-Start collecting weather data
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-To collect weather measurements and forecasts from the DarkSky API, there is a task you could run periodically, probably once per hour. Here is an example:
-
-.. code-block::
-
-   flexmeasures add external-weather-forecasts --location 33.4366,126.5269 --store-in-db 
-
-.. note::  DarkSky is not handing out tokens any more, as they have been bought by Apple (see `issue 3 <https://github.com/SeitaBV/flexmeasures/issues/56>`_).
-
-
-Preparing the job queue database and start workers
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-To let FlexMeasures queue forecasting and scheduling jobs, install a `Redis <https://redis.io/>`_ server and configure access to it within FlexMeasures' config file (see above). You can find the necessary settings in :ref:`redis-config`.
-
-Then run one worker for each kind of job (in a separate terminal):
-
-.. code-block::
-
-   flexmeasures run-worker --queue forecasting
-   flexmeasures run-worker --queue scheduling
-
-
-You can also clear the job queues:
-
-.. code-block::
-
-   flexmeasures clear-queue --queue forecasting
-   flexmeasures clear-queue --queue scheduling
-
-
-When the main FlexMeasures process runs (e.g. by ``flexmeasures run``\ ), the queues of forecasting and scheduling jobs can be visited at ``http://localhost:5000/tasks/forecasting`` and ``http://localhost:5000/tasks/schedules``\ , respectively (by admins).
+Install and configure Redis
+^^^^^^^^^^^^^^^^^^^^^^^
 
-When forecasts and schedules have been generated, they should be visible at ``http://localhost:5000/analytics``. You can also access forecasts via the FlexMeasures API at `GET  /api/v2_0/getPrognosis <api/v2_0.html#get--api-v2_0-getPrognosis>`_\ , and schedules via `GET  /api/v2_0/getDeviceMessage <api/v2_0.html#get--api-v2_0-getDeviceMessage>`_. 
+To let FlexMeasures queue forecasting and scheduling jobs, install a `Redis <https://redis.io/>`_ server (or rent one) and configure access to it within FlexMeasures' config file (see above). You can find the necessary settings in :ref:`redis-config`.

documentation/index.rst

@@ -45,6 +45,14 @@ The platform operator of FlexMeasures can be an Aggregator.
     concepts/security_auth
 
 
+.. toctree::
+    :caption: Tutorials
+    :maxdepth: 1
+
+    tut/posting_data
+    tut/forecasting_scheduling
+    tut/building_uis
+
 .. toctree::
     :caption: The in-built UI
     :maxdepth: 1
@@ -60,7 +68,6 @@ The platform operator of FlexMeasures can be an Aggregator.
     :maxdepth: 1
 
     api/introduction
-    api/simulation
     api/v2_0
     api/v1_3
     api/v1_2