Refactor scheduler interface - API and inner logic

documentation/api/notation.rst

@@ -162,6 +162,67 @@ For version 1, 2 and 3 of the API, only equidistant timeseries data is expected
 - "duration" should also be a multiple of the sensor resolution.
 
 
+.. _describing_flexibility:
+
+Describing flexibility
+^^^^^^^^^^^^^^^^^^^^^^^
+
+To compute a schedule, FlexMeasures needs to know state and context information which goes beyond
+the usual time series of the asset.
+
+We've grouped these two sources of information in two:
+
+Flex model
+""""""""""""
+
+The flexibility model describes to the scheduler what the flexible asset's state is,
+and what constraints or preferences should be taken into account.
+
+Which specific flexibility model is relevant in your scheduler usually relates to the sensor's asset type.
+Usually, not the whole flexibility model is needed (e.g. to be sent through the endpoint).
+For instance, FlexMeasures can infer missing values in the flex model, and even get them (as default) from the sensor's attributes.
+
+Here are the three flexibility models you can expect to be built-in:
+
+1) For storage sensors (e.g. battery, charge points), the schedule deals with the state of charge (SOC).
+    The possible flexibility parameters are:
+
+    - soc_at_start (defaults to 0)
+    - soc_unit (kWh or MWh)
+    - soc_min (defaults to 0)
+    - soc_max (defaults to max soc target)
+    - soc_targets (defaults to NaN values)
+    - roundtrip_efficiency (defaults to 100%)
+    - prefer_charging_sooner (defaults to True, also signals a preference to discharge later)
+
+  For some examples, see the `[POST] /sensors/(id)/schedule/trigger <../api/v3_0.html#post--api-v3_0-sensors-(id)-schedules-trigger>`_ endpoint docs.
+
+2) Shiftable process
+   .. todo:: A simple algorithm exists, needs integration into FlexMeasures and asset type clarified.
+
+3) Heat pumps TODO:
+   .. todo:: Also work in progress, needs model for heat loss compensation.
+
+In addition, folks who write their own custom scheduler (see :ref:`plugin_customization`) might also require their custom flexibility model.
+That's no problem, FlexMeasures will let the scheduler decide which flexibility model is relevant and to validate it. 
+
+.. note:: We also aim to model situations with more than one flexible asset, which can also have differing flexibility types.
+     This is ongoing architecture design work, and therefore happens in development settings, until we are happy 
+     with the outcomes. Thoughts welcome :) 
+
+
+Flex context
+"""""""""""""
+
+With the flexibility context, we aim to describe the EMS in which the sensor operates:
+
+- inflexible_device_sensors ― sensors which are relevant, but not flexible, e.g. solar
+- consumption_price_sensor ― the sensor which defines costs/revenues of consuming energy
+- production_price_sensor ― the sensor which defines cost/revenues of producing energy
+
+These should be independent on the asset type and consequently also do not depend on which scheduling algorithm is being used.
+
+
 .. _beliefs:
 
 Tracking the recording time of beliefs

documentation/changelog.rst

@@ -37,14 +37,16 @@ Infrastructure / Support
 * Remove bokeh dependency and obsolete UI views [see `PR #476 <http://www.github.com/FlexMeasures/flexmeasures/pull/476>`_]
 * Fix ``flexmeasures db-ops dump`` and ``flexmeasures db-ops restore`` not working in docker containers [see `PR #530 <http://www.github.com/FlexMeasures/flexmeasures/pull/530>`_] and incorrectly reporting a success when `pg_dump` and `pg_restore` are not installed [see `PR #526 <http://www.github.com/FlexMeasures/flexmeasures/pull/526>`_]
 * Plugins can save BeliefsSeries, too, instead of just BeliefsDataFrames [see `PR #523 <http://www.github.com/FlexMeasures/flexmeasures/pull/523>`_]
-* Improve documentation and code w.r.t. storage flexibility modelling ― prepare for handling other schedulers & merge battery and car charging schedulers [see `PR #511 <http://www.github.com/FlexMeasures/flexmeasures/pull/511>`_]
+* Improve documentation and code w.r.t. storage flexibility modelling ― prepare for handling other schedulers & merge battery and car charging schedulers [see `PR #511 <http://www.github.com/FlexMeasures/flexmeasures/pull/511>`_ and `PR #537 <http://www.github.com/FlexMeasures/flexmeasures/pull/537>`_]
 * Revised strategy for removing unchanged beliefs when saving data: retain the oldest measurement (ex-post belief), too [see `PR #518 <http://www.github.com/FlexMeasures/flexmeasures/pull/518>`_]
 * Scheduling test for maximizing self-consumption, and improved time series db queries for fixed tariffs (and other long-term constants) [see `PR #532 <http://www.github.com/FlexMeasures/flexmeasures/pull/532>`_]
 * Clean up table formatting for ``flexmeasures show`` CLI commands [see `PR #540 <http://www.github.com/FlexMeasures/flexmeasures/pull/540>`_]
 
 
 .. warning:: The CLI command ``flexmeasures monitor tasks`` has been renamed to ``flexmeasures monitor last-run``. The old name will stop working in version 0.13.
 
+.. warning:: The API endpoint (`[POST] /sensors/(id)/schedule/trigger <api/v3_0.html#post--api-v3_0-sensors-(id)-schedules-trigger>`_) and CLI command (``flexmeasures add schedule``) to make new schedules will (in v0.13) deprecate the storage flexibility parameters (they move to the ``flex-model`` parameter group), as well as the parameters describing other sensors (they move to ``flex-context``).
+
 
 v0.11.3 | November 2, 2022
 ============================

documentation/cli/change_log.rst

@@ -12,6 +12,7 @@ since v0.12.0 | November XX, 2022
 * Fix ``flexmeasures db-ops dump`` and ``flexmeasures db-ops restore`` incorrectly reporting a success when `pg_dump` and `pg_restore` are not installed.
 * Add ``flexmeasures monitor last-seen``. 
 * Rename ``flexmeasures monitor tasks`` to ``flexmeasures monitor last-run``. 
+* Rename ``flexmeasures add schedule`` to ``flexmeasures add schedule-for-storage`` (in expectation of more scheduling commands, based on in-built flex models). 
 
 since v0.11.0 | August 28, 2022
 ==============================

documentation/cli/commands.rst

@@ -34,7 +34,7 @@ of which some are referred to in this documentation.
 ``flexmeasures add sensor``                       Add a new sensor.
 ``flexmeasures add beliefs``                      Load beliefs from file.
 ``flexmeasures add forecasts``                    Create forecasts.
-``flexmeasures add schedule``                     Create a charging schedule.
+``flexmeasures add schedule-for-storage``         Create a charging schedule for a storage asset.
 ``flexmeasures add holidays``                     Add holiday annotations to accounts and/or assets.
 ``flexmeasures add annotation``                   Add annotation to accounts, assets and/or sensors.
 ``flexmeasures add toy-account``                  Create a toy account, for tutorials and trying things.

documentation/configuration.rst

@@ -247,6 +247,9 @@ Time to live for UDI event ids of successful scheduling jobs. Set a negative tim
 
 Default: ``timedelta(days=7)``
 
+
+.. _planning_horizon_config:
+
 FLEXMEASURES_PLANNING_HORIZON
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

documentation/dev/docker-compose.rst

@@ -96,7 +96,7 @@ Next, we put a scheduling job in the worker's queue. This only works because we
 
 .. code-block:: console
 
-    flexmeasures add schedule --sensor-id 2 --optimization-context-id 3 \
+    flexmeasures add schedule-for-storage --sensor-id 2 --optimization-context-id 3 \
         --start ${TOMORROW}T07:00+01:00 --duration PT12H --soc-at-start 50% \
         --roundtrip-efficiency 90% --as-job
 

documentation/index.rst

@@ -41,7 +41,7 @@ A tiny, but complete example: Let's install FlexMeasures from scratch. Then, usi
     $ flexmeasures db upgrade  # create tables
     $ flexmeasures add toy-account --kind battery  # setup account & a user, a battery (Id 2) and a market (Id 3)
     $ flexmeasures add beliefs --sensor-id 3 --source toy-user prices-tomorrow.csv --timezone utc  # load prices, also possible per API
-    $ flexmeasures add schedule --sensor-id 2 --consumption-price-sensor 3 \
+    $ flexmeasures add schedule-for-storage --sensor-id 2 --consumption-price-sensor 3 \
         --start ${TOMORROW}T07:00+01:00 --duration PT12H \
         --soc-at-start 50% --roundtrip-efficiency 90%  # this is also possible per API
     $ flexmeasures show beliefs --sensor-id 2 --start ${TOMORROW}T07:00:00+01:00 --duration PT12H  # also visible per UI, of course

documentation/plugin/customisation.rst

@@ -32,12 +32,8 @@ The following minimal example gives you an idea of some meta information you can
         __author__ = "My Company"
         __version__ = "2"
 
-        def schedule(
+        def compute_schedule(
             self,
-            sensor: Sensor,
-            start: datetime,
-            end: datetime,
-            resolution: timedelta,
             *args,
             **kwargs
         ):
@@ -49,12 +45,16 @@ The following minimal example gives you an idea of some meta information you can
                 sensor.get_attribute("capacity_in_mw"),
                 index=pd.date_range(start, end, freq=resolution, closed="left"),
             )
+
+        def inspect_config(self):
+            """Do not care about any flex config sent in."""
+            self.config_inspected = True
 
 
-.. note:: It's possible to add arguments that describe the asset flexibility and the EMS context in more detail. For example,
-          for storage assets we support various state-of-charge parameters. For now, the existing in-built schedulers are the best documentation.
-          We are working on documenting this better, so the learning curve becomes easier.
-
+.. note:: It's possible to add arguments that describe the asset flexibility model and the flexibility (EMS) context in more detail.
+          For example, for storage assets we support various state-of-charge parameters. For details on flexibility model and context,
+          see :ref:`describing_flexibility` and the `[POST] /sensors/(id)/schedule/trigger <../api/v3_0.html#post--api-v3_0-sensors-(id)-schedules-trigger>`_ endpoint.
+        
 
 Finally, make your scheduler be the one that FlexMeasures will use for certain sensors:
 

documentation/tut/forecasting_scheduling.rst

@@ -129,13 +129,13 @@ A second way to add scheduling jobs is via the CLI, so this is available for peo
 
 .. code-block:: console
 
-    flexmeasures add schedule --sensor-id 2 --optimization-context-id 3 \
+    flexmeasures add schedule-for-storage --sensor-id 2 --optimization-context-id 3 \
         --start 2022-07-05T07:00+01:00 --duration PT12H \
         --soc-at-start 50% --roundtrip-efficiency 90% --as-job
 
 Here, the ``--as-job`` parameter makes the difference for queueing ― without it, the schedule is computed right away.
 
-Run ``flexmeasures add schedule --help`` for more information.
+Run ``flexmeasures add schedule-for-storage --help`` for more information.
 
 
 .. _getting_prognoses:

documentation/tut/posting_data.rst

@@ -269,29 +269,32 @@ Posting flexibility states
 
 There is one more crucial kind of data that FlexMeasures needs to know about: What are the current states of flexible devices?
 For example, a battery has a certain state of charge, which is relevant to describe the flexibility that the battery currently has.
+In our terminology, this is called the "flex model" and you can read more at `ref`:describing_flexibility:.
 
-Owners of such devices can post these states along with triggering the creation of a new schedule, to `[POST] /schedules/trigger <../api/v3_0.html#post--api-v3_0-sensors-(id)-schedules-trigger>`_.
+Owners of such devices can post the flex model along with triggering the creation of a new schedule, to `[POST] /schedules/trigger <../api/v3_0.html#post--api-v3_0-sensors-(id)-schedules-trigger>`_.
 The URL might look like this:
 
 .. code-block:: html
 
     https://company.flexmeasures.io/api/<version>/sensors/10/schedules/trigger
 
-This example triggers a schedule for a power sensor (with ID 10) of a battery asset, asking to take into account the battery's current state of charge.
+The following example triggers a schedule for a power sensor (with ID 10) of a battery asset, asking to take into account the battery's current state of charge.
 From this, FlexMeasures derives the energy flexibility this battery has in the next 48 hours and computes an optimal charging schedule.
-The endpoint allows to limit the flexibility range and also to set target values.
+The endpoint also allows to limit the flexibility range and also to set target values.
 
 .. code-block:: json
 
         {
-            "value": 12.1,
-            "datetime": "2015-06-02T10:00:00+00:00",
-            "unit": "kWh"
+            "start": "2015-06-02T10:00:00+00:00",
+            "flex_model": {
+                "soc_at_start": 12.1,
+                "soc_unit": "kWh"
+            }
         }
 
 .. note:: At the moment, FlexMeasures only supports flexibility models suitable for batteries and car chargers here (asset types "battery", "one-way_evse" or "two-way_evse").
           This will be expanded to other flexible assets as needed.
 
-.. note:: Flexibility states are not persisted. To record a history of the state of charge, set up a separate sensor and post data to it using `[POST]  /sensors/data <../api/v3_0.html#post--api-v3_0-sensors-data>`_ (see :ref:`posting_sensor_data`).
+.. note:: Flexibility states are persisted on sensor attributes. To record a more complete history of the state of charge, set up a separate sensor and post data to it using `[POST]  /sensors/data <../api/v3_0.html#post--api-v3_0-sensors-data>`_ (see :ref:`posting_sensor_data`).
 
-In :ref:`how_queue_scheduling`, we'll cover what happens when FlexMeasurers is triggered to create a new schedule, and how those schedules can be retrieved via the API, so they can be used to steer assets.
+In :ref:`how_queue_scheduling`, we'll cover what happens when FlexMeasures is triggered to create a new schedule, and how those schedules can be retrieved via the API, so they can be used to steer assets.

documentation/tut/toy-example-from-scratch.rst

@@ -21,7 +21,7 @@ Below are the ``flexmeasures`` CLI commands we'll run, and which we'll explain s
     # load prices to optimise the schedule against
     $ flexmeasures add beliefs --sensor-id 3 --source toy-user prices-tomorrow.csv --timezone utc
     # make the schedule
-    $ flexmeasures add schedule --sensor-id 2 --consumption-price-sensor 3 \
+    $ flexmeasures add schedule-for-storage --sensor-id 2 --consumption-price-sensor 3 \
         --start ${TOMORROW}T07:00+01:00 --duration PT12H \
         --soc-at-start 50% --roundtrip-efficiency 90%
 
@@ -268,7 +268,7 @@ To keep it short, we'll only ask for a 12-hour window starting at 7am. Finally,
 
 .. code-block:: console
 
-    $ flexmeasures add schedule --sensor-id 2 --consumption-price-sensor 3 \
+    $ flexmeasures add schedule-for-storage --sensor-id 2 --consumption-price-sensor 3 \
         --start ${TOMORROW}T07:00+01:00 --duration PT12H \
         --soc-at-start 50% --roundtrip-efficiency 90%
     New schedule is stored.
@@ -314,4 +314,4 @@ We can also look at the charging schedule in the `FlexMeasures UI <http://localh
 Recall that we only asked for a 12 hour schedule here. We started our schedule *after* the high price peak (at 5am) and it also had to end *before* the second price peak fully realised (at 9pm). Our scheduler didn't have many opportunities to optimize, but it found some. For instance, it does buy at the lowest price (around 3pm) and sells it off when prices start rising again (around 6pm).
 
 
-.. note:: The ``flexmeasures add schedule`` command also accepts state-of-charge targets, so the schedule can be more sophisticated. But that is not the point of this tutorial. See ``flexmeasures add schedule --help``. 
+.. note:: The ``flexmeasures add schedule-for-storage`` command also accepts state-of-charge targets, so the schedule can be more sophisticated. But that is not the point of this tutorial. See ``flexmeasures add schedule-for-storage --help``. 

flexmeasures/api/common/responses.py

@@ -264,6 +264,15 @@ def unknown_schedule(message: str) -> ResponseTuple:
     return dict(result="Rejected", status="UNKNOWN_SCHEDULE", message=message), 400
 
 
+def invalid_flex_config(message: str) -> ResponseTuple:
+    return (
+        dict(
+            result="Rejected", status="UNPROCESSABLE_ENTITY", message=dict(json=message)
+        ),
+        422,
+    )
+
+
 @BaseMessage("The requested backup is not known.")
 def unrecognized_backup(message: str) -> ResponseTuple:
     return dict(result="Rejected", status="UNRECOGNIZED_BACKUP", message=message), 400

flexmeasures/api/v1_2/implementations.py

@@ -39,7 +39,6 @@
 )
 from flexmeasures.data.models.time_series import Sensor
 from flexmeasures.data.services.resources import has_assets, can_access_asset
-from flexmeasures.data.models.planning.utils import ensure_storage_specs
 from flexmeasures.utils.time_utils import duration_isoformat
 
 
@@ -98,17 +97,18 @@ def get_device_message_response(generic_asset_name_groups, duration):
             resolution = sensor.event_resolution
 
             # Schedule the asset
-            storage_specs = dict(
-                soc_at_start=sensor.generic_asset.get_attribute("soc_in_mwh"),
-                prefer_charging_sooner=False,
-            )
-            storage_specs = ensure_storage_specs(
-                storage_specs, sensor, start, end, resolution
+            scheduler = StorageScheduler(
+                sensor=sensor,
+                start=start,
+                end=end,
+                resolution=resolution,
+                flex_model=dict(
+                    soc_at_start=sensor.generic_asset.get_attribute("soc_in_mwh"),
+                    prefer_charging_sooner=False,
+                ),
             )
             try:
-                schedule = StorageScheduler().schedule(
-                    sensor, start, end, resolution, storage_specs=storage_specs
-                )
+                schedule = scheduler.compute_schedule()
             except UnknownPricesException:
                 return unknown_prices()
             except UnknownMarketException: