CLI: flexmeasures show beliefs (#379)

* first version of flexmeasures show beliefs Signed-off-by: Nicolas Höning <nicolas@seita.nl> * add types lib for mypy Signed-off-by: Nicolas Höning <nicolas@seita.nl> * construct title at the end when we have all info updated Signed-off-by: Nicolas Höning <nicolas@seita.nl> * update changelogs Signed-off-by: Nicolas Höning <nicolas@seita.nl> * add -belief-time-before option Signed-off-by: Nicolas Höning <nicolas@seita.nl> * add simple tests for all show data commands Signed-off-by: Nicolas Höning <nicolas@seita.nl> * flexmeasures/conftest.py fix typo in ficture name Signed-off-by: Nicolas Höning <nicolas@seita.nl> * verifying that CLI tests do not run on GithubActions Signed-off-by: Nicolas Höning <nicolas@seita.nl> * move back to ignoring CLI commands on Github Actions Signed-off-by: Nicolas Höning <nicolas@seita.nl> * small doc improvement Signed-off-by: Nicolas Höning <nicolas@seita.nl> * better handling of DEBUG var Signed-off-by: Nicolas Höning <nicolas@seita.nl> * improve docstring of two fixtures Signed-off-by: Nicolas Höning <nicolas@seita.nl> * if sensor share the same unit, still show it on the y-axis ; Also introduce Schema for DataSourceIdField Signed-off-by: Nicolas Höning <nicolas@seita.nl> * Use AwareDateTimeField for --start Signed-off-by: Nicolas Höning <nicolas@seita.nl> * also use schemas for --duration and --belief-time Signed-off-by: Nicolas Höning <nicolas@seita.nl> * use SensorIdField schema for --sensor-id Signed-off-by: Nicolas Höning <nicolas@seita.nl> * use schemas in show commands for account and asset Signed-off-by: Nicolas Höning <nicolas@seita.nl> * simplify possible traces we need to show with one_deterministic_belief_per_event=True Signed-off-by: Nicolas Höning <nicolas@seita.nl> * use source.description in title Signed-off-by: Nicolas Höning <nicolas@seita.nl> * improve in-code documentation Signed-off-by: Nicolas Höning <nicolas@seita.nl> * resample data from different sensors to minimal resolution if possible; be more informative if no data found for a sensor Signed-off-by: Nicolas Höning <nicolas@seita.nl> * simpler way of making sure data is resample to minimal resolution Signed-off-by: Nicolas Höning <nicolas@seita.nl>
FlexMeasures · Mar 8, 2022 · 8fe7e8b · 8fe7e8b
1 parent 8d7f00c
commit 8fe7e8b
Show file tree

Hide file tree

Showing 13 changed files with 323 additions and 27 deletions.
diff --git a/ci/run_mypy.sh b/ci/run_mypy.sh
@@ -1,7 +1,7 @@
 #!/bin/bash
 set -e
 pip install --upgrade 'mypy>=0.902'
-pip install types-pytz types-requests types-Flask types-click types-redis types-tzlocal types-python-dateutil types-setuptools
+pip install types-pytz types-requests types-Flask types-click types-redis types-tzlocal types-python-dateutil types-setuptools types-tabulate
 # We are checking python files which have type hints, and leave out bigger issues we made issues for
 # * data/scripts: We'll remove legacy code: https://trello.com/c/1wEnHOkK/7-remove-custom-data-scripts
 # * data/models and data/services: https://trello.com/c/rGxZ9h2H/540-makequery-call-signature-is-incoherent

diff --git a/documentation/changelog.rst b/documentation/changelog.rst
@@ -13,7 +13,8 @@ New features
 * Add CLI option to pass a data unit when reading in time series data from CSV, so data can automatically be converted to the sensor unit [see `PR #341 <http://www.github.com/FlexMeasures/flexmeasures/pull/341>`_]
 * Add CLI option to specify custom strings that should be interpreted as NaN values when reading in time series data from CSV [see `PR #357 <http://www.github.com/FlexMeasures/flexmeasures/pull/357>`_]
 * Add CLI commands ``flexmeasures add sensor``, ``flexmeasures add asset-type``, ``flexmeasures add beliefs`` (which were experimental features before) [see `PR #337 <http://www.github.com/FlexMeasures/flexmeasures/pull/337>`_]
-* Add CLI commands for showing data [see `PR #339 <http://www.github.com/FlexMeasures/flexmeasures/pull/339>`_]
+* Add CLI commands for showing organisational structure [see `PR #339 <http://www.github.com/FlexMeasures/flexmeasures/pull/339>`_]
+* Add a CLI command for showing time series data [see `PR #379 <http://www.github.com/FlexMeasures/flexmeasures/pull/379>`_]
 * Add CLI command for attaching annotations to assets: ``flexmeasures add holidays`` adds public holidays [see `PR #343 <http://www.github.com/FlexMeasures/flexmeasures/pull/343>`_]
 * Add CLI command for resampling existing sensor data to new resolution [see `PR #360 <http://www.github.com/FlexMeasures/flexmeasures/pull/360>`_]
 * Add CLI command to edit/add an attribute on an asset or sensor. [see `PR #380 <http://www.github.com/FlexMeasures/flexmeasures/pull/380>`_]
@@ -27,7 +28,6 @@ Bugfixes
 Infrastructure / Support
 ----------------------
 * Plugins can import common FlexMeasures classes (like ``Asset`` and ``Sensor``) from a central place, using ``from flexmeasures import Asset, Sensor`` [see `PR #354 <http://www.github.com/FlexMeasures/flexmeasures/pull/354>`_]
-
 * Adapt CLI command for entering some initial structure (``flexmeasures add structure``) to new datamodel [see `PR #349 <http://www.github.com/FlexMeasures/flexmeasures/pull/349>`_]
 * Align documentation requirements with pip-tools [see `PR #384 <http://www.github.com/FlexMeasures/flexmeasures/pull/384>`_]
 

diff --git a/documentation/cli/change_log.rst b/documentation/cli/change_log.rst
@@ -7,8 +7,9 @@ FlexMeasures CLI Changelog
 since v0.9.0 | January 26, 2022
 =====================
 
-* Add CLI commands for showing data ``flexmeasures show accounts``, ``flexmeasures show account``, ``flexmeasures show roles``, ``flexmeasures show asset-types``, ``flexmeasures show asset`` and ``flexmeasures show data-sources``.
-* Add ``flexmeasures edit resample-data`` CLI command to resample sensor data to a different resolution.
+
+* Add CLI commands for showing data ``flexmeasures show accounts``, ``flexmeasures show account``, ``flexmeasures show roles``, ``flexmeasures show asset-types``, ``flexmeasures show asset``, ``flexmeasures show data-sources``, and ``flexmeasures show beliefs``.
+* Add ``flexmeasures db-ops resample-data`` CLI command to resample sensor data to a different resolution.
 * Add ``flexmeasures edit attribute`` CLI command to edit/add an attribute on an asset or sensor.
 * Add ``flexmeasures add toy-account`` for tutorials and trying things.
 * Add ``flexmeasures add schedule`` to create a new schedule for a given power sensor.

diff --git a/documentation/cli/commands.rst b/documentation/cli/commands.rst
@@ -49,6 +49,7 @@ of which some are referred to in this documentation.
 ``flexmeasures show asset``                       Show an asset and its sensors.
 ``flexmeasures show roles``                       List available account- and user roles.
 ``flexmeasures show data-sources``                List available data sources.
+``flexmeasures show beliefs``                     Plot time series data.
 ================================================= =======================================
 
 

diff --git a/flexmeasures/cli/data_show.py b/flexmeasures/cli/data_show.py
@@ -1,15 +1,24 @@
 """CLI Tasks for listing database contents - most useful in development"""
 
+from typing import Optional, List
+from datetime import datetime, timedelta
+
 import click
 from flask import current_app as app
 from flask.cli import with_appcontext
 from tabulate import tabulate
 from humanize import naturaldelta, naturaltime
+import uniplot
 
 from flexmeasures.data.models.user import Account, AccountRole, User, Role
 from flexmeasures.data.models.data_sources import DataSource
 from flexmeasures.data.models.generic_assets import GenericAsset, GenericAssetType
-from flexmeasures.data.models.time_series import Sensor
+from flexmeasures.data.models.time_series import Sensor, TimedBelief
+from flexmeasures.data.schemas.generic_assets import GenericAssetIdField
+from flexmeasures.data.schemas.sensors import SensorIdField
+from flexmeasures.data.schemas.account import AccountIdField
+from flexmeasures.data.schemas.sources import DataSourceIdField
+from flexmeasures.data.schemas.times import AwareDateTimeField, DurationField
 
 
 @click.group("show")
@@ -72,16 +81,11 @@ def list_roles():
 
 @fm_show_data.command("account")
 @with_appcontext
-@click.option("--id", "account_id", type=int, required=True)
-def show_account(account_id):
+@click.option("--id", "account", type=AccountIdField(), required=True)
+def show_account(account):
     """
     Show information about an account, including users and assets.
     """
-    account: Account = Account.query.get(account_id)
-    if not account:
-        click.echo(f"No account with id {account_id} known.")
-        raise click.Abort
-
     click.echo(f"========{len(account.name) * '='}==========")
     click.echo(f"Account {account.name} (ID:{account.id}):")
     click.echo(f"========{len(account.name) * '='}==========\n")
@@ -94,7 +98,7 @@ def show_account(account_id):
         click.echo("Account has no roles.")
     click.echo()
 
-    users = User.query.filter_by(account_id=account_id).order_by(User.username).all()
+    users = User.query.filter_by(account_id=account.id).order_by(User.username).all()
     if not users:
         click.echo("No users in account ...")
     else:
@@ -115,7 +119,7 @@ def show_account(account_id):
 
     click.echo()
     assets = (
-        GenericAsset.query.filter_by(account_id=account_id)
+        GenericAsset.query.filter_by(account_id=account.id)
         .order_by(GenericAsset.name)
         .all()
     )
@@ -150,16 +154,11 @@ def list_asset_types():
 
 @fm_show_data.command("asset")
 @with_appcontext
-@click.option("--id", "asset_id", type=int, required=True)
-def show_generic_asset(asset_id):
+@click.option("--id", "asset", type=GenericAssetIdField(), required=True)
+def show_generic_asset(asset):
     """
     Show asset info and list sensors
     """
-    asset = GenericAsset.query.get(asset_id)
-    if not asset:
-        click.echo(f"No asset with id {asset_id} known.")
-        raise click.Abort
-
     click.echo(f"======{len(asset.name) * '='}==========")
     click.echo(f"Asset {asset.name} (ID:{asset.id}):")
     click.echo(f"======{len(asset.name) * '='}==========\n")
@@ -175,7 +174,7 @@ def show_generic_asset(asset_id):
 
     click.echo()
     sensors = (
-        Sensor.query.filter_by(generic_asset_id=asset_id).order_by(Sensor.name).all()
+        Sensor.query.filter_by(generic_asset_id=asset.id).order_by(Sensor.name).all()
     )
     if not sensors:
         click.echo("No sensors in asset ...")
@@ -218,4 +217,105 @@ def list_data_sources():
     )
 
 
+@fm_show_data.command("beliefs")
+@with_appcontext
+@click.option(
+    "--sensor-id",
+    "sensors",
+    required=True,
+    multiple=True,
+    type=SensorIdField(),
+    help="ID of sensor(s). This argument can be given multiple times.",
+)
+@click.option(
+    "--from",
+    "start",
+    type=AwareDateTimeField(),
+    required=True,
+    help="Plot starting at this datetime. Follow up with a timezone-aware datetime in ISO 6801 format.",
+)
+@click.option(
+    "--duration",
+    "duration",
+    type=DurationField(),
+    required=True,
+    help="Duration of the plot, after --from. Follow up with a duration in ISO 6801 format, e.g. PT1H (1 hour) or PT45M (45 minutes).",
+)
+@click.option(
+    "--belief-time-before",
+    "belief_time_before",
+    type=AwareDateTimeField(),
+    required=False,
+    help="Time at which beliefs had been known. Follow up with a timezone-aware datetime in ISO 6801 format.",
+)
+@click.option(
+    "--source-id",
+    "source",
+    required=False,
+    type=DataSourceIdField(),
+    help="Source of the beliefs (an existing source id).",
+)
+def plot_beliefs(
+    sensors: List[Sensor],
+    start: datetime,
+    duration: timedelta,
+    belief_time_before: Optional[datetime],
+    source: Optional[DataSource],
+):
+    """
+    Show a simple plot of belief data directly in the terminal.
+    """
+    sensors = list(sensors)
+    min_resolution = min([s.event_resolution for s in sensors])
+
+    # query data
+    beliefs_by_sensor = TimedBelief.search(
+        sensors=sensors,
+        event_starts_after=start,
+        event_ends_before=start + duration,
+        beliefs_before=belief_time_before,
+        source=source,
+        one_deterministic_belief_per_event=True,
+        resolution=min_resolution,
+        sum_multiple=False,
+    )
+    # only keep non-empty
+    for s in sensors:
+        if beliefs_by_sensor[s.name].empty:
+            click.echo(f"No data found for sensor '{s.name}' (Id: {s.id})")
+            beliefs_by_sensor.pop(s.name)
+            sensors.remove(s)
+    if len(beliefs_by_sensor.keys()) == 0:
+        click.echo("No data found!")
+        raise click.Abort()
+    first_df = beliefs_by_sensor[sensors[0].name]
+
+    # Build title
+    if len(sensors) == 1:
+        title = f"Beliefs for Sensor '{sensors[0].name}' (Id {sensors[0].id}).\n"
+    else:
+        title = f"Beliefs for Sensor(s) [{','.join([s.name for s in sensors])}], (Id(s): [{','.join([str(s.id) for s in sensors])}]).\n"
+    title += f"Data spans {naturaldelta(duration)} and starts at {start}."
+    if belief_time_before:
+        title += f"\nOnly beliefs made before: {belief_time_before}."
+    if source:
+        title += f"\nSource: {source.description}"
+    title += f"\nThe time resolution (x-axis) is {naturaldelta(min_resolution)}."
+
+    uniplot.plot(
+        [
+            beliefs.event_value
+            for beliefs in [beliefs_by_sensor[sn] for sn in [s.name for s in sensors]]
+        ],
+        title=title,
+        color=True,
+        lines=True,
+        y_unit=first_df.sensor.unit
+        if len(beliefs_by_sensor) == 1
+        or all(sensor.unit == first_df.sensor.unit for sensor in sensors)
+        else "",
+        legend_labels=[s.name for s in sensors],
+    )
+
+
 app.cli.add_command(fm_show_data)
diff --git a/flexmeasures/cli/tests/test_data_show.py b/flexmeasures/cli/tests/test_data_show.py
@@ -0,0 +1,112 @@
+import pytest
+
+from flexmeasures.data.models.time_series import Sensor
+
+
+@pytest.mark.skip_github
+def test_list_accounts(app, fresh_db, setup_accounts_fresh_db):
+    from flexmeasures.cli.data_show import list_accounts
+
+    runner = app.test_cli_runner()
+    result = runner.invoke(list_accounts)
+
+    assert "All accounts on this" in result.output
+    for account in setup_accounts_fresh_db.values():
+        assert account.name in result.output
+    assert result.exit_code == 0
+
+
+@pytest.mark.skip_github
+def test_list_roles(app, fresh_db, setup_roles_users_fresh_db):
+    from flexmeasures.cli.data_show import list_roles
+
+    runner = app.test_cli_runner()
+    result = runner.invoke(list_roles)
+
+    assert "Account roles" in result.output
+    assert "User roles" in result.output
+    for role in ("account-admin", "Supplier", "Dummy"):
+        assert role in result.output
+    assert result.exit_code == 0
+
+
+@pytest.mark.skip_github
+def test_list_asset_types(app, fresh_db, setup_generic_asset_types_fresh_db):
+    from flexmeasures.cli.data_show import list_asset_types
+
+    runner = app.test_cli_runner()
+    result = runner.invoke(list_asset_types)
+
+    for asset_type in setup_generic_asset_types_fresh_db.values():
+        assert asset_type.name in result.output
+    assert result.exit_code == 0
+
+
+@pytest.mark.skip_github
+def test_list_sources(app, fresh_db, setup_sources_fresh_db):
+    from flexmeasures.cli.data_show import list_data_sources
+
+    runner = app.test_cli_runner()
+    result = runner.invoke(list_data_sources)
+
+    for source in setup_sources_fresh_db.values():
+        assert source.name in result.output
+    assert result.exit_code == 0
+
+
+@pytest.mark.skip_github
+def test_show_accounts(app, fresh_db, setup_accounts_fresh_db):
+    from flexmeasures.cli.data_show import show_account
+
+    fresh_db.session.flush()  # get IDs in DB
+
+    runner = app.test_cli_runner()
+    result = runner.invoke(
+        show_account, ["--id", setup_accounts_fresh_db["Prosumer"].id]
+    )
+
+    assert "Account Test Prosumer Account" in result.output
+    assert "No users in account" in result.output
+    assert result.exit_code == 0
+
+
+@pytest.mark.skip_github
+def test_show_asset(app, fresh_db, setup_generic_assets_fresh_db):
+    from flexmeasures.cli.data_show import show_generic_asset
+
+    fresh_db.session.flush()  # get IDs in DB
+
+    runner = app.test_cli_runner()
+    result = runner.invoke(
+        show_generic_asset,
+        ["--id", setup_generic_assets_fresh_db["test_wind_turbine"].id],
+    )
+
+    assert "Asset Test wind turbine" in result.output
+    assert "No sensors in asset" in result.output
+    assert result.exit_code == 0
+
+
+@pytest.mark.skip_github
+def test_plot_beliefs(app, fresh_db, setup_beliefs_fresh_db):
+    from flexmeasures.cli.data_show import plot_beliefs
+
+    sensor = Sensor.query.filter(Sensor.name == "epex_da").one_or_none()
+
+    runner = app.test_cli_runner()
+    result = runner.invoke(
+        plot_beliefs,
+        [
+            "--sensor-id",
+            sensor.id,
+            "--from",
+            "2021-03-28T16:00+01",
+            "--duration",
+            "PT1H",
+        ],
+    )
+
+    assert "Beliefs for Sensor 'epex_da'" in result.output
+    assert "Data spans an hour" in result.output
+
+    assert result.exit_code == 0