CLI: flexmeasures show beliefs

ci/run_mypy.sh

@@ -1,7 +1,7 @@
 #!/bin/bash
 set -e
 pip install --upgrade 'mypy>=0.902'
-pip install types-pytz types-requests types-Flask types-click types-redis types-tzlocal types-python-dateutil types-setuptools
+pip install types-pytz types-requests types-Flask types-click types-redis types-tzlocal types-python-dateutil types-setuptools types-tabulate
 # We are checking python files which have type hints, and leave out bigger issues we made issues for
 # * data/scripts: We'll remove legacy code: https://trello.com/c/1wEnHOkK/7-remove-custom-data-scripts
 # * data/models and data/services: https://trello.com/c/rGxZ9h2H/540-makequery-call-signature-is-incoherent

documentation/changelog.rst

@@ -13,8 +13,8 @@ New features
 * Add CLI option to pass a data unit when reading in time series data from CSV, so data can automatically be converted to the sensor unit [see `PR #341 <http://www.github.com/FlexMeasures/flexmeasures/pull/341>`_]
 * Add CLI option to specify custom strings that should be interpreted as NaN values when reading in time series data from CSV [see `PR #357 <http://www.github.com/FlexMeasures/flexmeasures/pull/357>`_]
 * Add CLI commands ``flexmeasures add sensor``, ``flexmeasures add asset-type``, ``flexmeasures add beliefs`` (which were experimental features before). [see `PR #337 <http://www.github.com/FlexMeasures/flexmeasures/pull/337>`_]
-* Add CLI commands for showing data [see `PR #339 <http://www.github.com/FlexMeasures/flexmeasures/pull/339>`_]
-
+* Add CLI commands for showing structural data [see `PR #339 <http://www.github.com/FlexMeasures/flexmeasures/pull/339>`_]
+* Add a CLI command for showing time series data [see `PR #379 <http://www.github.com/FlexMeasures/flexmeasures/pull/379>`_]
 * Add CLI command for attaching annotations to assets: ``flexmeasures add holidays`` adds public holidays [see `PR #343 <http://www.github.com/FlexMeasures/flexmeasures/pull/343>`_]
 * Add CLI command for resampling existing sensor data to new resolution [see `PR #360 <http://www.github.com/FlexMeasures/flexmeasures/pull/360>`_]
 * Add CLI command to add a toy account for tutorials and trying things [see `PR #368 <http://www.github.com/FlexMeasures/flexmeasures/pull/368>`_].

documentation/cli/change_log.rst

@@ -7,7 +7,7 @@ FlexMeasures CLI Changelog
 since v0.9.0 | January 26, 2022
 =====================
 
-* Add CLI commands for showing data ``flexmeasures show accounts``, ``flexmeasures show account``, ``flexmeasures show roles``, ``flexmeasures show asset-types``, ``flexmeasures show asset`` and ``flexmeasures show data-sources``.
+* Add CLI commands for showing data ``flexmeasures show accounts``, ``flexmeasures show account``, ``flexmeasures show roles``, ``flexmeasures show asset-types``, ``flexmeasures show asset``, ``flexmeasures show data-sources``, and ``flexmeasures show beliefs``.
 * Add ``flexmeasures db-ops resample-data`` CLI command to resample sensor data to a different resolution.
 * Add ``flexmeasures add toy-account`` for tutorials and trying things.
 * Rename ``flexmeasures add structure`` to ``flexmeasures add initial-structure``. 

documentation/cli/commands.rst

@@ -48,6 +48,7 @@ of which some are referred to in this documentation.
 ``flexmeasures show asset``                       Show an asset and its sensors.
 ``flexmeasures show roles``                       List available account- and user roles.
 ``flexmeasures show data-sources``                List available data sources.
+``flexmeasures show beliefs``                     Plot time series data.
 ================================================= =======================================
 
 

documentation/index.rst

@@ -6,10 +6,10 @@ Planning ahead allows flexible assets to serve the whole system with their flexi
 e.g. by shifting or curtailing energy use.
 This can also be profitable for their owners.
 
-The *FlexMeasures Platform* is the intelligent backend to support real-time energy flexibility apps, rapidly and scalable. 
+> The *FlexMeasures EMS* is the intelligent backend to support real-time energy flexibility apps, rapidly and scalable. 
 
 - Developing energy flexibility services (e.g. to enable demand response) is crucial, but expensive.
-- FlexMeasures reduces development costs with real-time data integrations, uncertainty models and API/UI support.
+- FlexMeasures reduces development costs with real-time data intelligence & integrations, uncertainty models and API/UI support.
 
 As possible users, we see energy service companies (ESCOs) who want to build real-time apps & services around energy flexibility for their customers, or medium/large industrials who are looking for support in their internal digital tooling. However, even small companies and hobby projects might find FlexMeasures useful! 
 

flexmeasures/cli/data_show.py

@@ -1,15 +1,18 @@
 """CLI Tasks for listing database contents - most useful in development"""
 
+from typing import Optional, List
 import click
 from flask import current_app as app
 from flask.cli import with_appcontext
 from tabulate import tabulate
 from humanize import naturaldelta, naturaltime
+import isodate
+import uniplot
 
 from flexmeasures.data.models.user import Account, AccountRole, User, Role
 from flexmeasures.data.models.data_sources import DataSource
 from flexmeasures.data.models.generic_assets import GenericAsset, GenericAssetType
-from flexmeasures.data.models.time_series import Sensor
+from flexmeasures.data.models.time_series import Sensor, TimedBelief
 
 
 @click.group("show")
@@ -218,4 +221,117 @@ def list_data_sources():
     )
 
 
+@fm_show_data.command("beliefs")
+@with_appcontext
+@click.option(
+    "--sensor-id",
+    "sensor_ids",
+    type=int,
+    required=True,
+    multiple=True,
+    help="ID of sensor(s). This argument can be given multiple times.",
+)
+@click.option(
+    "--from",
+    "start_str",
+    required=True,
+    help="Plot starting at this datetime. Follow up with a timezone-aware datetime in ISO 6801 format.",
+)
+@click.option(
+    "--duration",
+    "duration_str",
+    required=True,
+    help="Duration of the plot, after start_str. Follow up with a duration in ISO 6801 format, e.g. PT1H (1 hour) or PT45M (45 minutes).",
+)
+@click.option(
+    "--belief-time-before",
+    "belief_time_before_str",
+    required=False,
+    help="Time at which beliefs had been known. Follow up with a timezone-aware datetime in ISO 6801 format.",
+)
+@click.option(
+    "--source-id",
+    "source_id",
+    required=False,
+    type=int,
+    help="Source of the beliefs (an existing source id).",
+)
+def plot_beliefs(
+    sensor_ids: List[int],
+    start_str: str,
+    duration_str: str,
+    belief_time_before_str: Optional[str],
+    source_id: Optional[int],
+):
+    """
+    Show a simple plot of belief data directly in the terminal.
+    """
+    # handle required params: sensor, start, duration
+    sensor_names: List[str] = []
+    for sensor_id in sensor_ids:
+        sensor: Sensor = Sensor.query.get(sensor_id)
+        if not sensor:
+            click.echo(f"No sensor with id {sensor_id} known.")
+            raise click.Abort
+        sensor_names.append(sensor.name)
+    start = isodate.parse_datetime(start_str)  # TODO: make sure it has a tz
+    duration = isodate.parse_duration(duration_str)
+    # handle belief time
+    belief_time_before = None
+    if belief_time_before_str:
+        belief_time_before = isodate.parse_datetime(belief_time_before_str)
+    # handle source
+    source: DataSource = None
+    if source_id:
+        source = DataSource.query.get(source_id)
+        if not source:
+            click.echo(f"No source with id {source_id} known.")
+            raise click.Abort
+    # query data
+    beliefs_by_sensor = TimedBelief.search(
+        sensors=list(sensor_ids),
+        event_starts_after=start,
+        event_ends_before=start + duration,
+        beliefs_before=belief_time_before,
+        source=source,
+        sum_multiple=False,
+    )
+    # only keep non-empty
+    beliefs_by_sensor = {
+        sensor_name: beliefs
+        for (sensor_name, beliefs) in beliefs_by_sensor.items()
+        if not beliefs.empty
+    }
+    if len(beliefs_by_sensor.keys()) == 0:
+        click.echo("No data found!")
+        raise click.Abort()
+    sensor_names = list(beliefs_by_sensor.keys())
+    first_df = list(beliefs_by_sensor.values())[0]
+
+    # Build title
+    if len(sensor_ids) == 1:
+        title = f"Beliefs for Sensor '{sensor_names[0]}' (Id {sensor_ids[0]}).\n"
+    else:
+        title = f"Beliefs for Sensor(s) [{','.join(sensor_names)}], (Id(s): [{','.join([str(sid) for sid in sensor_ids])}]).\n"
+    title += f"Data spans {naturaldelta(duration)} and starts at {start}."
+    if belief_time_before:
+        title += f"\nOnly beliefs made before: {belief_time_before}."
+    if source:
+        title += f"\nSource: {source.name}"
+    if len(beliefs_by_sensor.values()) == 1:
+        title += f"\nThe time resolution (x-axis) is {naturaldelta(first_df.sensor.event_resolution)}."
+
+    uniplot.plot(
+        [
+            beliefs.event_value
+            for beliefs in [beliefs_by_sensor[sn] for sn in sensor_names]
+        ],
+        title=title,
+        color=True,
+        lines=True,
+        y_unit=first_df.sensor.unit if len(beliefs_by_sensor.values()) == 1 else "",
+        legend_labels=sensor_names,
+    )
+
+
 app.cli.add_command(fm_show_data)

flexmeasures/cli/tests/test_data_show.py

@@ -0,0 +1,112 @@
+import pytest
+
+from flexmeasures.data.models.time_series import Sensor
+
+
+@pytest.mark.skip_github
+def test_list_accounts(app, fresh_db, setup_accounts_fresh_db):
+    from flexmeasures.cli.data_show import list_accounts
+
+    runner = app.test_cli_runner()
+    result = runner.invoke(list_accounts)
+
+    assert "All accounts on this" in result.output
+    for account in setup_accounts_fresh_db.values():
+        assert account.name in result.output
+    assert result.exit_code == 0
+
+
+@pytest.mark.skip_github
+def test_list_roles(app, fresh_db, setup_roles_users_fresh_db):
+    from flexmeasures.cli.data_show import list_roles
+
+    runner = app.test_cli_runner()
+    result = runner.invoke(list_roles)
+
+    assert "Account roles" in result.output
+    assert "User roles" in result.output
+    for role in ("account-admin", "Supplier", "Dummy"):
+        assert role in result.output
+    assert result.exit_code == 0
+
+
+@pytest.mark.skip_github
+def test_list_asset_types(app, fresh_db, setup_generic_asset_types_fresh_db):
+    from flexmeasures.cli.data_show import list_asset_types
+
+    runner = app.test_cli_runner()
+    result = runner.invoke(list_asset_types)
+
+    for asset_type in setup_generic_asset_types_fresh_db.values():
+        assert asset_type.name in result.output
+    assert result.exit_code == 0
+
+
+@pytest.mark.skip_github
+def test_list_sources(app, fresh_db, setup_sources_fresh_db):
+    from flexmeasures.cli.data_show import list_data_sources
+
+    runner = app.test_cli_runner()
+    result = runner.invoke(list_data_sources)
+
+    for source in setup_sources_fresh_db.values():
+        assert source.name in result.output
+    assert result.exit_code == 0
+
+
+@pytest.mark.skip_github
+def test_show_accounts(app, fresh_db, setup_accounts_fresh_db):
+    from flexmeasures.cli.data_show import show_account
+
+    fresh_db.session.flush()  # get IDs in DB
+
+    runner = app.test_cli_runner()
+    result = runner.invoke(
+        show_account, ["--id", setup_accounts_fresh_db["Prosumer"].id]
+    )
+
+    assert "Account Test Prosumer Account" in result.output
+    assert "No users in account" in result.output
+    assert result.exit_code == 0
+
+
+@pytest.mark.skip_github
+def test_show_asset(app, fresh_db, setup_generic_assets_fresh_db):
+    from flexmeasures.cli.data_show import show_generic_asset
+
+    fresh_db.session.flush()  # get IDs in DB
+
+    runner = app.test_cli_runner()
+    result = runner.invoke(
+        show_generic_asset,
+        ["--id", setup_generic_assets_fresh_db["test_wind_turbine"].id],
+    )
+
+    assert "Asset Test wind turbine" in result.output
+    assert "No sensors in asset" in result.output
+    assert result.exit_code == 0
+
+
+@pytest.mark.skip_github
+def test_plot_beliefs(app, fresh_db, setup_beliefs_fresh_db):
+    from flexmeasures.cli.data_show import plot_beliefs
+
+    sensor = Sensor.query.filter(Sensor.name == "epex_da").one_or_none()
+
+    runner = app.test_cli_runner()
+    result = runner.invoke(
+        plot_beliefs,
+        [
+            "--sensor-id",
+            sensor.id,
+            "--from",
+            "2021-03-28T16:00+01",
+            "--duration",
+            "PT1H",
+        ],
+    )
+
+    assert "Beliefs for Sensor 'epex_da'" in result.output
+    assert "Data spans an hour" in result.output
+
+    assert result.exit_code == 0

flexmeasures/conftest.py

@@ -247,6 +247,15 @@ def create_test_markets(db) -> Dict[str, Market]:
 
 @pytest.fixture(scope="module")
 def setup_sources(db) -> Dict[str, DataSource]:
+    return create_sources(db)
+
+
+@pytest.fixture(scope="function")
+def setup_sources_fresh_db(fresh_db) -> Dict[str, DataSource]:
+    return create_sources(fresh_db)
+
+
+def create_sources(db) -> Dict[str, DataSource]:
     seita_source = DataSource(name="Seita", type="demo script")
     db.session.add(seita_source)
     entsoe_source = DataSource(name="ENTSO-E", type="demo script")
@@ -417,7 +426,20 @@ def setup_assets(
 
 
 @pytest.fixture(scope="module")
-def setup_beliefs(db: SQLAlchemy, setup_markets, setup_sources) -> int:
+def setup_beliefs(db, setup_markets, setup_sources) -> int:
+    """Make some beliefs."""
+    return create_beliefs(db, setup_markets, setup_sources)
+
+
+@pytest.fixture(scope="function")
+def setup_beliefs_fresh_db(
+    fresh_db, setup_markets_fresh_db, setup_sources_fresh_db
+) -> int:
+    """Make some beliefs."""
+    return create_beliefs(fresh_db, setup_markets_fresh_db, setup_sources_fresh_db)
+
+
+def create_beliefs(db: SQLAlchemy, setup_markets, setup_sources) -> int:
     """
     :returns: the number of beliefs set up
     """

requirements/app.in

@@ -52,5 +52,6 @@ marshmallow>=3
 marshmallow-polyfield
 marshmallow-sqlalchemy>=0.23.1
 webargs
+uniplot
 # flask should be after all the flask plugins, because setup might find they ARE flask
 flask>=1.0

requirements/app.txt

@@ -24,8 +24,7 @@ attrs==21.4.0
 babel==2.9.1
     # via py-moneyed
 backports.zoneinfo==0.2.1
-    # via
-    #   workalendar
+    # via workalendar
 bcrypt==3.2.0
     # via -r requirements/app.in
 blinker==1.4
@@ -211,6 +210,7 @@ numpy==1.22.2
     #   statsmodels
     #   timely-beliefs
     #   timetomodel
+    #   uniplot
 openturns==1.18
     # via timely-beliefs
 outcome==1.1.0
@@ -370,6 +370,8 @@ trio-websocket==0.9.2
     # via selenium
 typing-extensions==4.1.1
     # via py-moneyed
+uniplot==0.5.0
+    # via -r requirements/app.in
 urllib3[secure]==1.26.8
     # via
     #   requests