481 extend documentation and toy tutorial with scheduling within the …

…context of a building with PV (#534)

Expand toy tutorial and chart functionality (incl. a more sensible use of the color encoding and data source attribute in charts), by:

- Expanding the toy tutorial with creation of solar sensor and solar forecasts from CSV
- Expanding the toy tutorial with a new schedule that takes into account solar
- Allowing nesting of sensors in the `sensors_to_show` attribute, to denote that multiple sensors should be shown together, and dealing with all of its implications (axis labels, use of colors, etc.)
- Visually distinguishing forecasts, schedules and measurements by source type rather than by belief horizon (incl. a simple db migration)
- Adding a CLI command to create a new DataSource (so that forecasts are visually presented as originating from a forecaster)



* fix charging sensor name in tutorial

Signed-off-by: F.N. Claessen <felix@seita.nl>

* update price sensor name in tutorial (prefer non-capitalized sensor names)

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Print out account ID if toy account already exists

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Fix for changed default when reading in timezone naive data: we no longer assume utc, but raise instead (see PR #521)

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add solar power sensor to toy account, and allow existing toy accounts to be updated by simply running `flexmeasures add toy-account` again

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Support JSON and Callable arguments

Signed-off-by: F.N. Claessen <felix@seita.nl>

* get or create price sensor

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Refactor to keep original order of sensor ids

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add CLI option to pass inflexible device sensors

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Expand tutorial with scheduling against solar, which limits the available headroom for the battery

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add note about resampling

Signed-off-by: F.N. Claessen <felix@seita.nl>

* black

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Allow nesting sensors_to_show to support layering multiple sensors in a single row within the multi-row asset chart

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Textual changes

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Remove print statement

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add missing timezone

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Fix call to make_schedule

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Ensure storage specs, also when make_schedule is called directly (i.e. no job)

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Legend shows which asset a sensor belongs to, too, and sensors that share a name don't share a color (note that sensors belonging to the same asset aren't allowed to have the same name)

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add CLI command for adding a data source

Signed-off-by: F.N. Claessen <felix@seita.nl>

* DB migration for source types 'forecaster' and 'scheduler'

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Attribute toy solar forecasts to a new DataSource of type 'forecaster'

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Check common cases of shared sensor types

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add version identifier and missing parenthesis

Signed-off-by: F.N. Claessen <felix@seita.nl>

* black and flake8

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Fix test (no legend in PositionFieldDef)

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Remove redundant detail encodings

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Upgrade timely-beliefs for required feature from SeitaBV/timely-beliefs#122

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Merge db revisions

Signed-off-by: F.N. Claessen <felix@seita.nl>

* black

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Speed up viz by avoiding redundant client-side transformation

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Bump vega-lite and vegaembed

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Ensure full-width bar width

Signed-off-by: F.N. Claessen <felix@seita.nl>

* black

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Update CLI scheduling command by adding `for-storage`

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Improve introduction to using `flexmeasures add source`

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Introduce follow-up solar example

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Refactor: rename "scheduling script" to "scheduler" and "forecasting script" to "forecaster"

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Clarify event start domain setting

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Raise instead of warn in case toy account already exists

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Clarify loop over kwargs

Signed-off-by: F.N. Claessen <felix@seita.nl>

* black

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Merge steps

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Clarify when we choose not to show the sensor name

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add return type annotations

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add default sensors_to_show attribute to test asset

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add test cases for bad sensors_to_show attribute

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Validate sensors_to_show attribute

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Simplify code lines

Signed-off-by: F.N. Claessen <felix@seita.nl>

* black

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Refactor: move util function

Signed-off-by: F.N. Claessen <felix@seita.nl>

* flake8

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Changelog entry plus upgrade instructions

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add documentation for nested sensors_to_show attribute

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Resolve import error

Signed-off-by: F.N. Claessen <felix@seita.nl>

* redundant space

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Resolve circular import

Signed-off-by: F.N. Claessen <felix@seita.nl>

* add a missing output style

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

* Fix GH Issue #604

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Correct spacing of header markings

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add sensors_to_show attribute to tutorial printout

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Correct sensor name in tutorial

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Correct column name in tutorial

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Shift sensor ids in tutorial

Signed-off-by: F.N. Claessen <felix@seita.nl>

* fix capitalization

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Update uniplot output (looks like later versions have a more intuitive choice of y-axis ticks)

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Correct data source IDs

Signed-off-by: F.N. Claessen <felix@seita.nl>

* test

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Revert "test"

This reverts commit b393cad.

* Resolve circular import coming to light through mypy

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Generalize util function

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add documentation for new CLI command

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add documentation for new CLI option

Signed-off-by: F.N. Claessen <felix@seita.nl>

---------

Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: Nicolas Höning <nicolas@seita.nl>
Co-authored-by: Nicolas Höning <nicolas@seita.nl>

documentation/changelog.rst

@@ -8,9 +8,12 @@ v0.13.0 | April XX, 2023
 
 .. warning:: The API endpoint (`[POST] /sensors/(id)/schedules/trigger <api/v3_0.html#post--api-v3_0-sensors-(id)-schedules-trigger>`_) to make new schedules sunsets the deprecated (since v0.12) storage flexibility parameters (they move to the ``flex-model`` parameter group), as well as the parameters describing other sensors (they move to ``flex-context``).
 
+.. warning:: Upgrading to this version requires running ``flexmeasures db upgrade`` (you can create a backup first with ``flexmeasures db-ops dump``).
+
 New features
 -------------
 * Keyboard control over replay [see `PR #562 <https://www.github.com/FlexMeasures/flexmeasures/pull/562>`_]
+* Overlay charts (e.g. power profiles) on the asset page using the `sensors_to_show` attribute, and distinguish plots by source (different trace), sensor (different color) and source type (different stroke dash) [see `PR #534 <https://www.github.com/FlexMeasures/flexmeasures/pull/534>`_]
 * The ``FLEXMEASURES_MAX_PLANNING_HORIZON`` config setting can also be set as an integer number of planning steps rather than just as a fixed duration, which makes it possible to schedule further ahead in coarser time steps [see `PR #583 <https://www.github.com/FlexMeasures/flexmeasures/pull/583>`_]
 * Different text styles for CLI output for errors, warnings or success messages. [see `PR #609 <https://www.github.com/FlexMeasures/flexmeasures/pull/609>`_]
 

documentation/cli/change_log.rst

@@ -4,6 +4,12 @@
 FlexMeasures CLI Changelog
 **********************
 
+since v0.13.0 | April XX, 2023
+=================================
+
+* Add ``flexmeasures add source`` CLI command for adding a new data source.
+* Add ``--inflexible-device-sensor`` option to ``flexmeasures add schedule``.
+
 since v0.12.0 | January 04, 2023
 =================================
 

documentation/cli/commands.rst

@@ -33,6 +33,7 @@ of which some are referred to in this documentation.
 ``flexmeasures add asset``                        Create a new asset.
 ``flexmeasures add sensor``                       Add a new sensor.
 ``flexmeasures add beliefs``                      Load beliefs from file.
+``flexmeasures add source``                       Add a new data source.
 ``flexmeasures add forecasts``                    Create forecasts.
 ``flexmeasures add schedule for-storage``         Create a charging schedule for a storage asset.
 ``flexmeasures add holidays``                     Add holiday annotations to accounts and/or assets.

documentation/index.rst

@@ -39,12 +39,12 @@ A tiny, but complete example: Let's install FlexMeasures from scratch. Then, usi
     $ docker pull postgres; docker run --name pg-docker -e POSTGRES_PASSWORD=docker -e POSTGRES_DB=flexmeasures-db -d -p 5433:5432 postgres:latest 
     $ export SQLALCHEMY_DATABASE_URI="postgresql://postgres:docker@127.0.0.1:5433/flexmeasures-db" && export SECRET_KEY=notsecret 
     $ flexmeasures db upgrade  # create tables
-    $ flexmeasures add toy-account --kind battery  # setup account & a user, a battery (Id 2) and a market (Id 3)
-    $ flexmeasures add beliefs --sensor-id 3 --source toy-user prices-tomorrow.csv --timezone utc  # load prices, also possible per API
-    $ flexmeasures add schedule for-storage --sensor-id 2 --consumption-price-sensor 3 \
+    $ flexmeasures add toy-account --kind battery  # setup account incl. a user, battery (ID 1) and market (ID 2)
+    $ flexmeasures add beliefs --sensor-id 2 --source toy-user prices-tomorrow.csv --timezone utc  # load prices, also possible per API
+    $ flexmeasures add schedule for-storage --sensor-id 1 --consumption-price-sensor 2 \
         --start ${TOMORROW}T07:00+01:00 --duration PT12H \
         --soc-at-start 50% --roundtrip-efficiency 90%  # this is also possible per API
-    $ flexmeasures show beliefs --sensor-id 2 --start ${TOMORROW}T07:00:00+01:00 --duration PT12H  # also visible per UI, of course
+    $ flexmeasures show beliefs --sensor-id 1 --start ${TOMORROW}T07:00:00+01:00 --duration PT12H  # also visible per UI, of course
 
 We discuss this in more depth at :ref:`tut_toy_schedule`.
 
@@ -94,7 +94,7 @@ Your journey, from dipping your toes in the water towards being a happy FlexMeas
 
 
 
-Where to start reading ?
+Where to start reading?
 --------------------------
 
 You (the reader) might be a user connecting with a FlexMeasures server or working on hosting FlexMeasures. Maybe you are planning to develop a plugin or even core functionality. In :ref:`getting_started`, we have some helpful tips how to dive into this documentation!

documentation/tut/forecasting_scheduling.rst

@@ -58,7 +58,7 @@ In FlexMeasures, the usual way of creating forecasting jobs would be right in th
 So technically, you don't have to do anything to keep fresh forecasts.
 
 The decision which horizons to forecast is currently also taken by FlexMeasures. For power data, FlexMeasures makes this decision depending on the asset resolution. For instance, a resolution of 15 minutes leads to forecast horizons of 1, 6, 24 and 48 hours. For price data, FlexMeasures chooses to forecast prices forward 24 and 48 hours
-These are decent defaults, and fixing them has the advantage that scheduling scripts (see below) will know what to expect. However, horizons will probably become more configurable in the near future of FlexMeasures. 
+These are decent defaults, and fixing them has the advantage that schedulers (see below) will know what to expect. However, horizons will probably become more configurable in the near future of FlexMeasures.
 
 You can also add forecasting jobs directly via the CLI. We explain this practice in the next section.