Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

481 extend documentation and toy tutorial with scheduling within the context of a building with PV #534

Merged
Show file tree
Hide file tree
Changes from 44 commits
Commits
Show all changes
88 commits
Select commit Hold shift + click to select a range
31afb2e
fix charging sensor name in tutorial
Flix6x Nov 15, 2022
e60f97d
update price sensor name in tutorial (prefer non-capitalized sensor n…
Flix6x Nov 15, 2022
625bc2b
Print out account ID if toy account already exists
Flix6x Nov 15, 2022
fd19381
Fix for changed default when reading in timezone naive data: we no lo…
Flix6x Nov 15, 2022
a5278de
Add solar power sensor to toy account, and allow existing toy account…
Flix6x Nov 15, 2022
4220d92
Support JSON and Callable arguments
Flix6x Nov 15, 2022
07f8dc8
get or create price sensor
Flix6x Nov 15, 2022
bd977b2
Refactor to keep original order of sensor ids
Flix6x Nov 15, 2022
be49c47
Add CLI option to pass inflexible device sensors
Flix6x Nov 15, 2022
9320a6d
Expand tutorial with scheduling against solar, which limits the avail…
Flix6x Nov 15, 2022
639dd25
Add note about resampling
Flix6x Nov 15, 2022
e48e055
black
Flix6x Nov 15, 2022
644cdc2
Merge remote-tracking branch 'origin/main' into 481-extend-documentat…
Flix6x Nov 21, 2022
24cf156
Merge branch 'main' into 481-extend-documentation-and-toy-tutorial-wi…
Flix6x Nov 29, 2022
ba4ca70
Allow nesting sensors_to_show to support layering multiple sensors in…
Flix6x Nov 30, 2022
1b95b9d
Textual changes
Flix6x Nov 30, 2022
bd91967
Remove print statement
Flix6x Nov 30, 2022
cf16ecf
Add missing timezone
Flix6x Nov 30, 2022
7dcb001
Fix call to make_schedule
Flix6x Nov 30, 2022
141a794
Ensure storage specs, also when make_schedule is called directly (i.e…
Flix6x Nov 30, 2022
fec7cbb
Legend shows which asset a sensor belongs to, too, and sensors that s…
Flix6x Nov 30, 2022
15a59a2
Add CLI command for adding a data source
Flix6x Nov 30, 2022
42523e4
DB migration for source types 'forecaster' and 'scheduler'
Flix6x Nov 30, 2022
8d283d5
Attribute toy solar forecasts to a new DataSource of type 'forecaster'
Flix6x Nov 30, 2022
5170616
Check common cases of shared sensor types
Flix6x Nov 30, 2022
daf0273
Add version identifier and missing parenthesis
Flix6x Nov 30, 2022
cd4f41b
black and flake8
Flix6x Nov 30, 2022
9eb668d
Fix test (no legend in PositionFieldDef)
Flix6x Nov 30, 2022
71e983f
Remove redundant detail encodings
Flix6x Nov 30, 2022
f174c35
Upgrade timely-beliefs for required feature from https://github.com/S…
Flix6x Dec 2, 2022
b6a923c
Merge branch 'main' into 481-extend-documentation-and-toy-tutorial-wi…
Flix6x Dec 2, 2022
0d3e004
Merge branch 'main' into 481-extend-documentation-and-toy-tutorial-wi…
Flix6x Dec 12, 2022
a495bd1
Merge db revisions
Flix6x Dec 12, 2022
401cd3e
black
Flix6x Dec 12, 2022
491982e
Merge remote-tracking branch 'origin/main' into 481-extend-documentat…
Flix6x Dec 16, 2022
f67096f
Merge remote-tracking branch 'origin/main' into 481-extend-documentat…
Flix6x Dec 20, 2022
a337269
Speed up viz by avoiding redundant client-side transformation
Flix6x Dec 20, 2022
8992ea4
Bump vega-lite and vegaembed
Flix6x Dec 20, 2022
9159126
Merge branch 'main' into 481-extend-documentation-and-toy-tutorial-wi…
Flix6x Dec 20, 2022
4001deb
Ensure full-width bar width
Flix6x Dec 20, 2022
b2746ae
black
Flix6x Dec 21, 2022
5cae176
Merge branch 'main' into 481-extend-documentation-and-toy-tutorial-wi…
Flix6x Dec 30, 2022
42b55ee
Merge remote-tracking branch 'origin/main' into 481-extend-documentat…
Flix6x Jan 3, 2023
0583d83
Merge branch 'main' into 481-extend-documentation-and-toy-tutorial-wi…
Flix6x Jan 4, 2023
0df0632
Merge branch 'main' into 481-extend-documentation-and-toy-tutorial-wi…
Flix6x Jan 16, 2023
786255c
Merge branch 'main' into 481-extend-documentation-and-toy-tutorial-wi…
Flix6x Mar 9, 2023
061c153
Update CLI scheduling command by adding `for-storage`
Flix6x Mar 9, 2023
f1dc65d
Improve introduction to using `flexmeasures add source`
Flix6x Mar 9, 2023
5f99e63
Introduce follow-up solar example
Flix6x Mar 9, 2023
1c04f5f
Refactor: rename "scheduling script" to "scheduler" and "forecasting …
Flix6x Mar 9, 2023
9340613
Clarify event start domain setting
Flix6x Mar 9, 2023
67ad86c
Raise instead of warn in case toy account already exists
Flix6x Mar 9, 2023
7c96949
Clarify loop over kwargs
Flix6x Mar 21, 2023
6c25137
black
Flix6x Mar 21, 2023
ab000f2
Merge steps
Flix6x Mar 21, 2023
8b6c2c4
Clarify when we choose not to show the sensor name
Flix6x Mar 21, 2023
452e3c4
Add return type annotations
Flix6x Mar 24, 2023
6c76e93
Add default sensors_to_show attribute to test asset
Flix6x Mar 24, 2023
51d1068
Add test cases for bad sensors_to_show attribute
Flix6x Mar 24, 2023
8c1d013
Validate sensors_to_show attribute
Flix6x Mar 24, 2023
a592109
Merge remote-tracking branch 'origin/main' into 481-extend-documentat…
Flix6x Mar 24, 2023
1ae0b43
Simplify code lines
Flix6x Mar 24, 2023
770fef6
black
Flix6x Mar 24, 2023
0611afd
Refactor: move util function
Flix6x Mar 24, 2023
c16b01b
flake8
Flix6x Mar 27, 2023
838d9f3
Changelog entry plus upgrade instructions
Flix6x Mar 27, 2023
1a4008a
Add documentation for nested sensors_to_show attribute
Flix6x Mar 27, 2023
6f5c727
Resolve import error
Flix6x Mar 27, 2023
5e00455
redundant space
Flix6x Mar 27, 2023
b37fea8
Resolve circular import
Flix6x Mar 27, 2023
cbf8e17
add a missing output style
nhoening Mar 30, 2023
3d42a6f
Fix GH Issue #604
Flix6x Apr 5, 2023
f90ded6
Correct spacing of header markings
Flix6x Apr 9, 2023
810c6ea
Add sensors_to_show attribute to tutorial printout
Flix6x Apr 9, 2023
fca5f6e
Correct sensor name in tutorial
Flix6x Apr 9, 2023
eee7b97
Correct column name in tutorial
Flix6x Apr 9, 2023
625f189
Shift sensor ids in tutorial
Flix6x Apr 9, 2023
1467fc6
fix capitalization
Flix6x Apr 9, 2023
1f54b81
Update uniplot output (looks like later versions have a more intuitiv…
Flix6x Apr 9, 2023
4dc0361
Correct data source IDs
Flix6x Apr 9, 2023
b393cad
test
Flix6x Apr 9, 2023
a336a07
Merge remote-tracking branch 'origin/main' into HEAD
Flix6x Apr 9, 2023
d0f509b
Revert "test"
Flix6x Apr 9, 2023
57e61a7
Resolve circular import coming to light through mypy
Flix6x Apr 9, 2023
f98304d
Generalize util function
Flix6x Apr 9, 2023
dc58d9a
Add documentation for new CLI command
Flix6x Apr 10, 2023
deedb7d
Add documentation for new CLI option
Flix6x Apr 10, 2023
5d5481a
Merge remote-tracking branch 'origin/main' into 481-extend-documentat…
Flix6x Apr 12, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
150 changes: 105 additions & 45 deletions documentation/tut/toy-example-from-scratch.rst
Expand Up @@ -19,7 +19,7 @@ Below are the ``flexmeasures`` CLI commands we'll run, and which we'll explain s
# setup an account with a user, a battery (Id 2) and a market (Id 3)
$ flexmeasures add toy-account --kind battery
# load prices to optimise the schedule against
$ flexmeasures add beliefs --sensor-id 3 --source toy-user prices-tomorrow.csv --timezone utc
$ flexmeasures add beliefs --sensor-id 3 --source toy-user prices-tomorrow.csv --timezone Europe/Amsterdam
# make the schedule
$ flexmeasures add schedule for-storage --sensor-id 2 --consumption-price-sensor 3 \
--start ${TOMORROW}T07:00+01:00 --duration PT12H \
Expand Down Expand Up @@ -108,8 +108,9 @@ FlexMeasures offers a command to create a toy account with a battery:
$ flexmeasures add toy-account --kind battery

Toy account Toy Account with user toy-user@flexmeasures.io created successfully. You might want to run `flexmeasures show account --id 1`
The sensor for battery (dis)charging is <Sensor 2: discharging, unit: MW res.: 0:15:00>.
The sensor for Day ahead prices is <Sensor 3: Day ahead prices, unit: EUR/MWh res.: 1:00:00>.
The sensor recording battery power is <Sensor 2: discharging, unit: MW res.: 0:15:00>.
The sensor recording day-ahead prices is <Sensor 3: day-ahead prices, unit: EUR/MWh res.: 1:00:00>.
The sensor recording solar forecasts is <Sensor 4: production, unit: MW res.: 0:15:00>.

And with that, we're done with the structural data for this tutorial!

Expand Down Expand Up @@ -211,42 +212,42 @@ This is time series data, in FlexMeasures we call "beliefs". Beliefs can also be

.. code-block:: console

$ flexmeasures add beliefs --sensor-id 3 --source toy-user prices-tomorrow.csv --timezone utc
$ flexmeasures add beliefs --sensor-id 3 --source toy-user prices-tomorrow.csv --timezone Europe/Amsterdam
Successfully created beliefs

In FlexMeasures, all beliefs have a data source. Here, we use the username of the user we created earlier. We could also pass a user ID, or the name of a new data source we want to use for CLI scripts.

.. note:: Attention: We created and imported prices where the times have no time zone component! That happens a lot. Here, we localized the data to UTC time. So if you are in Amsterdam time, the start time for the first price, when expressed in your time zone, is actually `2022-03-03 01:00:00+01:00`.
.. note:: Attention: We created and imported prices where the times have no time zone component! That happens a lot. FlexMeasures can localize them for you to a given timezone. Here, we localized the data to the timezone of the price sensor - ``Europe/Amsterdam`` - so the start time for the first price is `2022-03-03 00:00:00+01:00` (midnight in Amsterdam).

Let's look at the price data we just loaded:

.. code-block:: console

$ flexmeasures show beliefs --sensor-id 3 --start ${TOMORROW}T01:00:00+01:00 --duration PT24H
Beliefs for Sensor 'Day ahead prices' (Id 3).
Data spans a day and starts at 2022-03-03 01:00:00+01:00.
$ flexmeasures show beliefs --sensor-id 3 --start ${TOMORROW}T00:00:00+01:00 --duration PT24H
Beliefs for Sensor 'day-ahead prices' (Id 3).
Data spans a day and starts at 2022-03-03 00:00:00+01:00.
The time resolution (x-axis) is an hour.
┌────────────────────────────────────────────────────────────┐
│ ▗▀▚▖ │ 18EUR/MWh
│ ▞ ▝▌ │
│ ▐ ▚ │
│ ▗▘ ▐ │
│ ▌ ▌ ▖ │
│ ▞ ▚ ▗▄▀▝▄ │
│ ▞ ▝▌ │
│ ▐ ▚ │
│ ▗▘ ▐ │
│ ▌ ▌ ▖ │
│ ▞ ▚ ▗▄▀▝▄ │
│ ▗▘ ▐ ▗▞▀ ▚ │ 13EUR/MWh
│ ▗▄▘ ▌ ▐▘ ▚ │
│ ▗▞▘ ▚ ▌ ▚ │
│▞▘ ▝▄ ▗ ▐ ▝▖ │
│ ▚▄▄▀▚▄▄ ▞▘▚ ▌ ▝▖ │
│ ▀▀▛ ▚ ▐ ▚ │
│ ▗▄▘ ▌ ▐▘ ▚ │
│ ▗▞▘ ▚ ▌ ▚ │
│▞▘ ▝▄ ▗ ▐ ▝▖ │
│ ▚▄▄▀▚▄▄ ▞▘▚ ▌ ▝▖ │
│ ▀▀▛ ▚ ▐ ▚ │
│ ▚ ▗▘ ▚│ 8EUR/MWh
│ ▌ ▗▘ ▝│
│ ▝▖ ▞ │
│ ▐▖ ▗▀ │
│ ▝▚▄▄▄▄▘ │
│ ▌ ▗▘ ▝│
│ ▝▖ ▞ │
│ ▐▖ ▗▀ │
│ ▝▚▄▄▄▄▘ │
└────────────────────────────────────────────────────────────┘
5 10 15 20
██ Day ahead prices
5 10 15 20
██ day-ahead prices



Expand Down Expand Up @@ -282,26 +283,26 @@ Great. Let's see what we made:
Data spans 12 hours and starts at 2022-03-04 07:00:00+01:00.
The time resolution (x-axis) is 15 minutes.
┌────────────────────────────────────────────────────────────┐
▐▀▀▌ ▛▀▀│
▞▌ ▞ ▐ ▌ │ 0.4MW
▌▌ ▌ ▐ ▐ │
▗▘▌ ▌ ▐ ▐ │
▐ ▐ ▗▘ ▝▖ ▐ │
▞ ▐ │ 0.2MW
▗▘ ▐ ▌ │
▐ ▝▖ ▞ │
│▀▘───▀▀▀▀▀▀▀▀▀▀▀▀▀▀▌────▐─────▝▀▀▀▀▀▀▀▀─────▐▀▀▀▀▀▀▀▀▀─────│ 0MW
▗▘
▗▘ ▝▖ │ -0.2MW
▗▘
▌ ▞ ▌ ▐
▌ ▌ ▐ ▐ │ -0.4MW
▙▄▄ ▐▄▄
▐▌ ▐▀▀▌ ▛▀▀│
▐▌ ▞ ▚ ▌ │ 0.4MW
▌▌ ▌ ▐ ▗▘ │
▌▚ ▌ ▐ ▐ │
▗▘▐ ▗▘ ▐ ▐ │
▐ ▐ │ 0.2MW
▌ ▐ ▌ │
▗▘ ▌ ▌ │
│▀▀▀▀▀▀▀───▀▀▀▀▌─────▌────▝▀▀▀▀▀▀▀▀─────▐▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▘───│ 0MW
▗▘
▗▘
│ -0.2MW
▗▘
▝▖ ▞ ▌ ▐
▌ ▌ ▚ ▐ │ -0.4MW
│ ▙▄▄ ▐▄▄
└────────────────────────────────────────────────────────────┘
10 20 30 40
██ discharging
10 20 30 40
██ discharging


Here, negative values denote output from the grid, so that's when the battery gets charged.
Expand All @@ -311,7 +312,66 @@ We can also look at the charging schedule in the `FlexMeasures UI <http://localh
.. image:: https://github.com/FlexMeasures/screenshots/raw/main/tut/toy-schedule/sensor-data-charging.png
:align: center

Recall that we only asked for a 12 hour schedule here. We started our schedule *after* the high price peak (at 5am) and it also had to end *before* the second price peak fully realised (at 9pm). Our scheduler didn't have many opportunities to optimize, but it found some. For instance, it does buy at the lowest price (around 3pm) and sells it off when prices start rising again (around 6pm).
Recall that we only asked for a 12 hour schedule here. We started our schedule *after* the high price peak (at 4am) and it also had to end *before* the second price peak fully realised (at 8pm). Our scheduler didn't have many opportunities to optimize, but it found some. For instance, it does buy at the lowest price (at 2pm) and sells it off at the highest price within the given 12 hours (at 6pm).


.. note:: The ``flexmeasures add schedule for-storage`` command also accepts state-of-charge targets, so the schedule can be more sophisticated. But that is not the point of this tutorial. See ``flexmeasures add schedule for-storage --help``.
.. note:: The ``flexmeasures add schedule for-storage`` command also accepts state-of-charge targets, so the schedule can be more sophisticated. But that is not the point of this tutorial. See ``flexmeasures add schedule for-storage --help``.


Take into account solar production
---------------------------------------

Flix6x marked this conversation as resolved.
Show resolved Hide resolved
First, we'll create a new csv file with solar forecasts (MW, see the setup for sensor 4 above) for tomorrow.

.. code-block:: console

$ TOMORROW=$(date --date="next day" '+%Y-%m-%d')
$ echo "Hour,Price
$ ${TOMORROW}T00:00:00,0.0
$ ${TOMORROW}T01:00:00,0.0
$ ${TOMORROW}T02:00:00,0.0
$ ${TOMORROW}T03:00:00,0.0
$ ${TOMORROW}T04:00:00,0.01
$ ${TOMORROW}T05:00:00,0.03
$ ${TOMORROW}T06:00:00,0.06
$ ${TOMORROW}T07:00:00,0.1
$ ${TOMORROW}T08:00:00,0.14
$ ${TOMORROW}T09:00:00,0.17
$ ${TOMORROW}T10:00:00,0.19
$ ${TOMORROW}T11:00:00,0.21
$ ${TOMORROW}T12:00:00,0.22
$ ${TOMORROW}T13:00:00,0.21
$ ${TOMORROW}T14:00:00,0.19
$ ${TOMORROW}T15:00:00,0.17
$ ${TOMORROW}T16:00:00,0.14
$ ${TOMORROW}T17:00:00,0.1
$ ${TOMORROW}T18:00:00,0.06
$ ${TOMORROW}T19:00:00,0.03
$ ${TOMORROW}T20:00:00,0.01
$ ${TOMORROW}T21:00:00,0.0
$ ${TOMORROW}T22:00:00,0.0
$ ${TOMORROW}T23:00:00,0.0" > solar-tomorrow.csv

Then, we register a new forecaster and read in the created CSV file as beliefs data:
Flix6x marked this conversation as resolved.
Show resolved Hide resolved

.. code-block:: console

$ flexmeasures add source --name "toy-forecaster" --type forecaster
Added source <Data source 2 (toy-forecaster)>
Flix6x marked this conversation as resolved.
Show resolved Hide resolved
$ flexmeasures add beliefs --sensor-id 4 --source 2 solar-tomorrow.csv --timezone Europe/Amsterdam
Flix6x marked this conversation as resolved.
Show resolved Hide resolved
Successfully created beliefs

The one-hour CSV data is automatically resampled to the 15-minute resolution of the sensor that is recording solar production.

.. note:: The ``flexmeasures add beliefs`` command has many options to make sure the read-in data is correctly interpreted (unit, timezone, delimiter, etc). But that is not the point of this tutorial. See ``flexmeasures add beliefs --help``.

Now, we'll reschedule the battery while taking into account the solar production. This will have an effect on the available headroom for the battery.

.. code-block:: console

$ flexmeasures add schedule --sensor-id 2 --consumption-price-sensor 3 \
Flix6x marked this conversation as resolved.
Show resolved Hide resolved
--inflexible-device-sensor 4 \
--start ${TOMORROW}T07:00+01:00 --duration PT12H \
--soc-at-start 50% --roundtrip-efficiency 90%
New schedule is stored.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like us to show the schedule, if it is different, and explain the difference. (if it's not different, this section is not making too much sense).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The schedule is different (see top chart in this PR description), but I'd like to move this to a new issue, because it is not yet the chart I intend to show. The chart I intend to show requires our envisioned KPI reporting feature.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Running the tutorial with a fresh DB creates different assets to those shown in the tutorial.

Creating Toy Account

Input:

flexmeasures add toy-account --kind battery

Output:

  Created <GenericAsset None: 'toy-battery' (battery)>
  Created discharging
  Adding transmission zone type ...
  Adding NL transmission zone ...
  Created day-ahead prices
  Created <GenericAsset None: 'toy-solar' (solar)>
  Created production
  Toy account Toy Account with user toy-user@flexmeasures.io created successfully. You might want to run `flexmeasures show account --id 1`
  The sensor recording battery discharging is discharging (ID: 2).
  The sensor recording day-ahead prices is day-ahead prices (ID: 3).
  The sensor recording solar forecasts is production (ID: 4).

Showing account

Input:

flexmeasures show account --id 1

Output

===========================
Account Toy Account (ID: 1)
===========================

Account has no roles.

All users:
 
  ID  Name      Email                     Last Login    Roles
----  --------  ------------------------  ------------  -------------
   1  toy-user  toy-user@flexmeasures.io  None          account-admin

All assets:
 
  ID  Name         Type     Location
----  -----------  -------  -----------------
   1  toy-battery  battery  (52.374, 4.88969)
   3  toy-solar    solar    (52.374, 4.88969)

In the tutorial, here appears toy-building, as well.

3 changes: 2 additions & 1 deletion flexmeasures/api/v3_0/assets.py
Expand Up @@ -15,6 +15,7 @@
from flexmeasures.data.schemas.generic_assets import GenericAssetSchema as AssetSchema
from flexmeasures.api.common.schemas.generic_assets import AssetIdField
from flexmeasures.api.common.schemas.users import AccountIdField
from flexmeasures.utils.coding_utils import flatten_unique
from flexmeasures.ui.utils.view_utils import set_time_range_for_session


Expand Down Expand Up @@ -309,5 +310,5 @@ def get_chart_data(self, id: int, asset: GenericAsset, **kwargs):

Data for use in charts (in case you have the chart specs already).
"""
sensors = asset.sensors_to_show
sensors = flatten_unique(asset.sensors_to_show)
return asset.search_beliefs(sensors=sensors, as_json=True, **kwargs)