FlexMeasures · Flix6x · Dec 29, 2022 · Nov 23, 2022 · Nov 23, 2022 · Nov 23, 2022
diff --git a/documentation/api/change_log.rst b/documentation/api/change_log.rst
@@ -5,6 +5,24 @@ API change log
 
 .. note:: The FlexMeasures API follows its own versioning scheme. This is also reflected in the URL, allowing developers to upgrade at their own pace.
 
+v3.0-5 | 2022-12-30
+"""""""""""""""""""
+
+- Introduced ``flex-model`` and ``flex-context`` fields to `/sensors/<id>/schedules/trigger` (POST). They are dictionaries and group pre-existing fields:
+
+    - ``soc-at-start`` -> send in ``flex-model`` instead
+    - ``soc-min`` -> send in ``flex-model`` instead
+    - ``soc-max`` -> send in ``flex-model`` instead
+    - ``soc-unit`` -> send in ``flex-model`` instead
+    - ``roundtrip-efficiency`` -> send in ``flex-model`` instead
+    - ``prefer-charging-sooner`` -> send in ``flex-model`` instead
+    - ``consumption-price-sensor`` -> send in ``flex-context`` instead
+    - ``production-price-sensor`` -> send in ``flex-context`` instead
+    - ``inflexible-device-sensors`` -> send in ``flex-context`` instead
+
+- The field ``doc-sensor-id`` in `/sensors/<id>/schedules/trigger` (POST) has been deprecated.
+
+
 v3.0-4 | 2022-12-08
 """""""""""""""""""
 

diff --git a/documentation/api/notation.rst b/documentation/api/notation.rst
@@ -155,6 +155,78 @@ For version 1, 2 and 3 of the API, only equidistant timeseries data is expected
 - "duration" should also be a multiple of the sensor resolution.
 
 
+.. _describing_flexibility:
+
+Describing flexibility
+^^^^^^^^^^^^^^^^^^^^^^^
+
+FlexMeasures computes schedules for energy systems that consist of multiple devices that consume and/or produce electricity.
+We model a device as an asset with a power sensor, and compute schedules only for flexible devices, while taking into account inflexible devices.
+
+To compute a schedule, FlexMeasures first needs to assess under what configuration flexibility can be made available to the system.
+This is described by the `flex model` (information about the state and possible actions of the flexible device) and the `flex-context`
+(information about the system as a whole, in order to assess the value of activating flexibility).
+
+This information goes beyond the usual time series recorded by an asset's sensors. It's being sent through the API when triggering schedule computation.
+Some parts of it can be persisted on the asset & sensor model as attributes (that's design work in progress). 
+
+We distinguish the information with two groups:
+
+Flex model
+""""""""""""
+
+The flexibility model describes to the scheduler what the flexible asset's state is,
+and what constraints or preferences should be taken into account.
+Which type of flexibility model is relevant to a scheduler usually relates to the type of device.
+
+Usually, not the whole flexibility model is needed.
+FlexMeasures can infer missing values in the flex model, and even get them (as default) from the sensor's attributes.
+This means that API and CLI users don't have to send the whole flex model every time.
+
+Here are the three types of flexibility models you can expect to be built-in:
+
+1) For storage devices (e.g. batteries, charge points, electric vehicle batteries connected to charge points), the schedule deals with the state of charge (SOC).
+
+    The possible flexibility parameters are:
+
+    - ``soc-at-start`` (defaults to 0)
+    - ``soc-unit`` (kWh or MWh)
+    - ``soc-min`` (defaults to 0)
+    - ``soc-max`` (defaults to max soc target)
+    - ``soc-targets`` (defaults to NaN values)
+    - ``roundtrip-efficiency`` (defaults to 100%)
+    - ``prefer-charging-sooner`` (defaults to True, also signals a preference to discharge later)
+
+  For some examples, see the `[POST] /sensors/(id)/schedules/trigger <../api/v3_0.html#post--api-v3_0-sensors-(id)-schedules-trigger>`_ endpoint docs.
+
+2) Shiftable process
+
+   .. todo:: A simple algorithm exists, needs integration into FlexMeasures and asset type clarified.
+
+3) Heat pumps
+
+   .. todo:: Also work in progress, needs model for heat loss compensation.
+
+In addition, folks who write their own custom scheduler (see :ref:`plugin_customization`) might also require their custom flexibility model.
+That's no problem, FlexMeasures will let the scheduler decide which flexibility model is relevant and how it should be validated. 
+
+.. note:: We also aim to model situations with more than one flexible asset, with different types of flexibility.
+     This is ongoing architecture design work, and therefore happens in development settings, until we are happy 
+     with the outcomes. Thoughts welcome :) 
+
+
+Flex context
+"""""""""""""
+
+With the flexibility context, we aim to describe the system in which the flexible assets operates:
+
+- ``inflexible-device-sensors`` ― power sensors that are relevant, but not flexible, such as a sensor recording rooftop solar power connected behind the main meter, whose production falls under the same contract as the flexible device(s) being scheduled
+- ``consumption-price-sensor`` ― the sensor which defines costs/revenues of consuming energy
+- ``production-price-sensor`` ― the sensor which defines cost/revenues of producing energy
+
+These should be independent on the asset type and consequently also do not depend on which scheduling algorithm is being used.
+
+
 .. _beliefs:
 
 Tracking the recording time of beliefs

diff --git a/documentation/changelog.rst b/documentation/changelog.rst
@@ -40,14 +40,19 @@ Infrastructure / Support
 * Remove bokeh dependency and obsolete UI views [see `PR #476 <http://www.github.com/FlexMeasures/flexmeasures/pull/476>`_]
 * Fix ``flexmeasures db-ops dump`` and ``flexmeasures db-ops restore`` not working in docker containers [see `PR #530 <http://www.github.com/FlexMeasures/flexmeasures/pull/530>`_] and incorrectly reporting a success when `pg_dump` and `pg_restore` are not installed [see `PR #526 <http://www.github.com/FlexMeasures/flexmeasures/pull/526>`_]
 * Plugins can save BeliefsSeries, too, instead of just BeliefsDataFrames [see `PR #523 <http://www.github.com/FlexMeasures/flexmeasures/pull/523>`_]
-* Improve documentation and code w.r.t. storage flexibility modelling ― prepare for handling other schedulers & merge battery and car charging schedulers [see `PR #511 <http://www.github.com/FlexMeasures/flexmeasures/pull/511>`_]
+* Improve documentation and code w.r.t. storage flexibility modelling ― prepare for handling other schedulers & merge battery and car charging schedulers [see `PR #511 <http://www.github.com/FlexMeasures/flexmeasures/pull/511>`_ and `PR #537 <http://www.github.com/FlexMeasures/flexmeasures/pull/537>`_]
 * Revised strategy for removing unchanged beliefs when saving data: retain the oldest measurement (ex-post belief), too [see `PR #518 <http://www.github.com/FlexMeasures/flexmeasures/pull/518>`_]
 * Scheduling test for maximizing self-consumption, and improved time series db queries for fixed tariffs (and other long-term constants) [see `PR #532 <http://www.github.com/FlexMeasures/flexmeasures/pull/532>`_]
 * Clean up table formatting for ``flexmeasures show`` CLI commands [see `PR #540 <http://www.github.com/FlexMeasures/flexmeasures/pull/540>`_]
 * Add  ``"Deprecation"`` and ``"Sunset"`` response headers for API users of deprecated API versions, and log warnings for FlexMeasures hosts when users still use them [see `PR #554 <http://www.github.com/FlexMeasures/flexmeasures/pull/554>`_]
 * Explain how to avoid potential ``SMTPRecipientsRefused`` errors when using FlexMeasures in combination with a mail server [see `PR #558 <http://www.github.com/FlexMeasures/flexmeasures/pull/558>`_]
 
-.. warning:: The CLI command ``flexmeasures monitor tasks`` has been renamed to ``flexmeasures monitor last-run``. The old name will stop working in version 0.13.
+.. warning:: The API endpoint (`[POST] /sensors/(id)/schedules/trigger <api/v3_0.html#post--api-v3_0-sensors-(id)-schedules-trigger>`_) to make new schedules will (in v0.13) sunset the storage flexibility parameters (they move to the ``flex-model`` parameter group), as well as the parameters describing other sensors (they move to ``flex-context``).
+
+.. warning:: The CLI command ``flexmeasures monitor tasks`` has been  deprecated (it's being renamed to ``flexmeasures monitor last-run``). The old name will be sunset in version 0.13.
+
+.. warning:: The CLI command  ``flexmeasures add schedule`` has been renamed to ``flexmeasures add schedule for-storage``. The old name will be sunset in version 0.13.
+
 
 
 v0.11.3 | November 2, 2022

diff --git a/documentation/cli/change_log.rst b/documentation/cli/change_log.rst
@@ -12,6 +12,7 @@ since v0.12.0 | November XX, 2022
 * Fix ``flexmeasures db-ops dump`` and ``flexmeasures db-ops restore`` incorrectly reporting a success when `pg_dump` and `pg_restore` are not installed.
 * Add ``flexmeasures monitor last-seen``. 
 * Rename ``flexmeasures monitor tasks`` to ``flexmeasures monitor last-run``. 
+* Rename ``flexmeasures add schedule`` to ``flexmeasures add schedule for-storage`` (in expectation of more scheduling commands, based on in-built flex models). 
 
 since v0.11.0 | August 28, 2022
 ==============================

diff --git a/documentation/cli/commands.rst b/documentation/cli/commands.rst
@@ -34,7 +34,7 @@ of which some are referred to in this documentation.
 ``flexmeasures add sensor``                       Add a new sensor.
 ``flexmeasures add beliefs``                      Load beliefs from file.
 ``flexmeasures add forecasts``                    Create forecasts.
-``flexmeasures add schedule``                     Create a charging schedule.
+``flexmeasures add schedule for-storage``         Create a charging schedule for a storage asset.
 ``flexmeasures add holidays``                     Add holiday annotations to accounts and/or assets.
 ``flexmeasures add annotation``                   Add annotation to accounts, assets and/or sensors.
 ``flexmeasures add toy-account``                  Create a toy account, for tutorials and trying things.

diff --git a/documentation/configuration.rst b/documentation/configuration.rst
@@ -247,6 +247,9 @@ Time to live for UDI event ids of successful scheduling jobs. Set a negative tim
 
 Default: ``timedelta(days=7)``
 
+
+.. _planning_horizon_config:
+
 FLEXMEASURES_PLANNING_HORIZON
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

diff --git a/documentation/dev/docker-compose.rst b/documentation/dev/docker-compose.rst
@@ -96,7 +96,7 @@ Next, we put a scheduling job in the worker's queue. This only works because we
 
 .. code-block:: console
 
-    flexmeasures add schedule --sensor-id 2 --optimization-context-id 3 \
+    flexmeasures add schedule for-storage --sensor-id 2 --optimization-context-id 3 \
         --start ${TOMORROW}T07:00+01:00 --duration PT12H --soc-at-start 50% \
         --roundtrip-efficiency 90% --as-job
 

diff --git a/documentation/index.rst b/documentation/index.rst
@@ -41,7 +41,7 @@ A tiny, but complete example: Let's install FlexMeasures from scratch. Then, usi
     $ flexmeasures db upgrade  # create tables
     $ flexmeasures add toy-account --kind battery  # setup account & a user, a battery (Id 2) and a market (Id 3)
     $ flexmeasures add beliefs --sensor-id 3 --source toy-user prices-tomorrow.csv --timezone utc  # load prices, also possible per API
-    $ flexmeasures add schedule --sensor-id 2 --consumption-price-sensor 3 \
+    $ flexmeasures add schedule for-storage --sensor-id 2 --consumption-price-sensor 3 \
         --start ${TOMORROW}T07:00+01:00 --duration PT12H \
         --soc-at-start 50% --roundtrip-efficiency 90%  # this is also possible per API
     $ flexmeasures show beliefs --sensor-id 2 --start ${TOMORROW}T07:00:00+01:00 --duration PT12H  # also visible per UI, of course

diff --git a/documentation/plugin/customisation.rst b/documentation/plugin/customisation.rst
@@ -32,12 +32,8 @@ The following minimal example gives you an idea of some meta information you can
         __author__ = "My Company"
         __version__ = "2"
 
-        def schedule(
+        def compute_schedule(
             self,
-            sensor: Sensor,
-            start: datetime,
-            end: datetime,
-            resolution: timedelta,
             *args,
             **kwargs
         ):
@@ -49,12 +45,16 @@ The following minimal example gives you an idea of some meta information you can
                 sensor.get_attribute("capacity_in_mw"),
                 index=pd.date_range(start, end, freq=resolution, closed="left"),
             )
+
+        def inspect_config(self):
+            """Do not care about any flex config sent in."""
+            self.config_inspected = True
 
 
-.. note:: It's possible to add arguments that describe the asset flexibility and the EMS context in more detail. For example,
-          for storage assets we support various state-of-charge parameters. For now, the existing in-built schedulers are the best documentation.
-          We are working on documenting this better, so the learning curve becomes easier.
-
+.. note:: It's possible to add arguments that describe the asset flexibility model and the flexibility (EMS) context in more detail.
+          For example, for storage assets we support various state-of-charge parameters. For details on flexibility model and context,
+          see :ref:`describing_flexibility` and the `[POST] /sensors/(id)/schedules/trigger <../api/v3_0.html#post--api-v3_0-sensors-(id)-schedules-trigger>`_ endpoint.
+        
 
 Finally, make your scheduler be the one that FlexMeasures will use for certain sensors:
 

diff --git a/documentation/tut/forecasting_scheduling.rst b/documentation/tut/forecasting_scheduling.rst
@@ -101,22 +101,24 @@ There are two ways to queue a scheduling job:
 First, we can add a scheduling job to the queue via the API.
 We already learned about the `[POST] /schedules/trigger <../api/v3_0.html#post--api-v3_0-sensors-(id)-schedules-trigger>`_ endpoint in :ref:`posting_flex_states`, where we saw how to post a flexibility state (in this case, the state of charge of a battery at a certain point in time).
 
-Here, we extend that example with an additional target value, representing a desired future state of charge.
+Here, we extend that (storage) example with an additional target value, representing a desired future state of charge.
 
 .. code-block:: json
 
     {
-        "value": 12.1,
-        "datetime": "2015-06-02T10:00:00+00:00",
-        "unit": "kWh",
-        "targets": [
-            {
-                "value": 25,
-                "datetime": "2015-06-02T16:00:00+00:00"
-            }
-        ]
+        "start": "2015-06-02T10:00:00+00:00",
+        "flex-model": {
+            "soc-at-start": 12.1,
+            "soc-unit": "kWh"
+            "soc-targets": [
+                {
+                    "value": 25,
+                    "datetime": "2015-06-02T16:00:00+00:00"
+                }
+        }
     }
 
+
 We now have described the state of charge at 10am to be ``12.1``. In addition, we requested that it should be ``25`` at 4pm.
 For instance, this could mean that a car should be charged at 90% at that time.
 
@@ -129,13 +131,13 @@ A second way to add scheduling jobs is via the CLI, so this is available for peo
 
 .. code-block:: console
 
-    flexmeasures add schedule --sensor-id 2 --optimization-context-id 3 \
+    flexmeasures add schedule for-storage --sensor-id 2 --optimization-context-id 3 \
         --start 2022-07-05T07:00+01:00 --duration PT12H \
         --soc-at-start 50% --roundtrip-efficiency 90% --as-job
 
 Here, the ``--as-job`` parameter makes the difference for queueing ― without it, the schedule is computed right away.
 
-Run ``flexmeasures add schedule --help`` for more information.
+Run ``flexmeasures add schedule for-storage --help`` for more information.
 
 
 .. _getting_prognoses:

diff --git a/documentation/tut/posting_data.rst b/documentation/tut/posting_data.rst
@@ -269,29 +269,32 @@ Posting flexibility states
 
 There is one more crucial kind of data that FlexMeasures needs to know about: What are the current states of flexible devices?
 For example, a battery has a certain state of charge, which is relevant to describe the flexibility that the battery currently has.
+In our terminology, this is called the "flex model" and you can read more at :ref:`describing_flexibility`.
 
-Owners of such devices can post these states along with triggering the creation of a new schedule, to `[POST] /schedules/trigger <../api/v3_0.html#post--api-v3_0-sensors-(id)-schedules-trigger>`_.
+Owners of such devices can post the flex model along with triggering the creation of a new schedule, to `[POST] /schedules/trigger <../api/v3_0.html#post--api-v3_0-sensors-(id)-schedules-trigger>`_.
 The URL might look like this:
 
 .. code-block:: html
 
     https://company.flexmeasures.io/api/<version>/sensors/10/schedules/trigger
 
-This example triggers a schedule for a power sensor (with ID 10) of a battery asset, asking to take into account the battery's current state of charge.
+The following example triggers a schedule for a power sensor (with ID 10) of a battery asset, asking to take into account the battery's current state of charge.
 From this, FlexMeasures derives the energy flexibility this battery has in the next 48 hours and computes an optimal charging schedule.
-The endpoint allows to limit the flexibility range and also to set target values.
+The endpoint also allows to limit the flexibility range and also to set target values.
 
 .. code-block:: json
 
         {
-            "value": 12.1,
-            "datetime": "2015-06-02T10:00:00+00:00",
-            "unit": "kWh"
+            "start": "2015-06-02T10:00:00+00:00",
+            "flex-model": {
+                "soc-at-start": 12.1,
+                "soc-unit": "kWh"
+            }
         }
 
 .. note:: At the moment, FlexMeasures only supports flexibility models suitable for batteries and car chargers here (asset types "battery", "one-way_evse" or "two-way_evse").
           This will be expanded to other flexible assets as needed.
 
-.. note:: Flexibility states are not persisted. To record a history of the state of charge, set up a separate sensor and post data to it using `[POST]  /sensors/data <../api/v3_0.html#post--api-v3_0-sensors-data>`_ (see :ref:`posting_sensor_data`).
+.. note:: Flexibility states are persisted on sensor attributes. To record a more complete history of the state of charge, set up a separate sensor and post data to it using `[POST]  /sensors/data <../api/v3_0.html#post--api-v3_0-sensors-data>`_ (see :ref:`posting_sensor_data`).
 
-In :ref:`how_queue_scheduling`, we'll cover what happens when FlexMeasurers is triggered to create a new schedule, and how those schedules can be retrieved via the API, so they can be used to steer assets.
+In :ref:`how_queue_scheduling`, we'll cover what happens when FlexMeasures is triggered to create a new schedule, and how those schedules can be retrieved via the API, so they can be used to steer assets.
diff --git a/documentation/tut/toy-example-from-scratch.rst b/documentation/tut/toy-example-from-scratch.rst
@@ -21,7 +21,7 @@ Below are the ``flexmeasures`` CLI commands we'll run, and which we'll explain s
     # load prices to optimise the schedule against
     $ flexmeasures add beliefs --sensor-id 3 --source toy-user prices-tomorrow.csv --timezone utc
     # make the schedule
-    $ flexmeasures add schedule --sensor-id 2 --consumption-price-sensor 3 \
+    $ flexmeasures add schedule for-storage --sensor-id 2 --consumption-price-sensor 3 \
         --start ${TOMORROW}T07:00+01:00 --duration PT12H \
         --soc-at-start 50% --roundtrip-efficiency 90%
 
@@ -268,7 +268,7 @@ To keep it short, we'll only ask for a 12-hour window starting at 7am. Finally,
 
 .. code-block:: console
 
-    $ flexmeasures add schedule --sensor-id 2 --consumption-price-sensor 3 \
+    $ flexmeasures add schedule for-storage --sensor-id 2 --consumption-price-sensor 3 \
         --start ${TOMORROW}T07:00+01:00 --duration PT12H \
         --soc-at-start 50% --roundtrip-efficiency 90%
     New schedule is stored.
@@ -314,4 +314,4 @@ We can also look at the charging schedule in the `FlexMeasures UI <http://localh
 Recall that we only asked for a 12 hour schedule here. We started our schedule *after* the high price peak (at 5am) and it also had to end *before* the second price peak fully realised (at 9pm). Our scheduler didn't have many opportunities to optimize, but it found some. For instance, it does buy at the lowest price (around 3pm) and sells it off when prices start rising again (around 6pm).
 
 
-.. note:: The ``flexmeasures add schedule`` command also accepts state-of-charge targets, so the schedule can be more sophisticated. But that is not the point of this tutorial. See ``flexmeasures add schedule --help``. 
+.. note:: The ``flexmeasures add schedule for-storage`` command also accepts state-of-charge targets, so the schedule can be more sophisticated. But that is not the point of this tutorial. See ``flexmeasures add schedule for-storage --help``.