Issue 435 display sensor charts on asset page (#449)

Put sensor data to the forefront of the asset page. The form for editing the asset (which used to be front and center) has been moved to a new sidepanel. * Show updated sensor data and annotations together Signed-off-by: F.N. Claessen <felix@seita.nl> * Show spinner while fetching new data Signed-off-by: F.N. Claessen <felix@seita.nl> * Switch from id-based styling to class-based styling Signed-off-by: F.N. Claessen <felix@seita.nl> * Move styling to css, and lower spinner Signed-off-by: F.N. Claessen <felix@seita.nl> * Simplify and streamline datepicker fontsize Signed-off-by: F.N. Claessen <felix@seita.nl> * Streamline datepicker margins Signed-off-by: F.N. Claessen <felix@seita.nl> * Add margins and side panel activated on hover Signed-off-by: F.N. Claessen <felix@seita.nl> * Correct margins and padding of side panel to allow for custom ranges at the bottom of the calendar Signed-off-by: F.N. Claessen <felix@seita.nl> * Show single month Signed-off-by: F.N. Claessen <felix@seita.nl> * Fewer custom ranges Signed-off-by: F.N. Claessen <felix@seita.nl> * Side panel rounded similar to buttons rather than similar to cards Signed-off-by: F.N. Claessen <felix@seita.nl> * Align box shadows of cards and calendar Signed-off-by: F.N. Claessen <felix@seita.nl> * Non-transparent cards Signed-off-by: F.N. Claessen <felix@seita.nl> * Simplified padding notation Signed-off-by: F.N. Claessen <felix@seita.nl> * Move chart actions buttons away from the card's corner (negative margin) Signed-off-by: F.N. Claessen <felix@seita.nl> * Rotate y-axis labels to improve legibility Signed-off-by: F.N. Claessen <felix@seita.nl> * Remove sensor chart title if the same information is already contained in the y-axis label Signed-off-by: F.N. Claessen <felix@seita.nl> * Move unit to right side of tooltip Signed-off-by: F.N. Claessen <felix@seita.nl> * Style predefined datetime ranges Signed-off-by: F.N. Claessen <felix@seita.nl> * Raise column to top without requiring flex display Signed-off-by: F.N. Claessen <felix@seita.nl> * Rename sidepanel class and separate styling specific to the sidepanel being on the left Signed-off-by: F.N. Claessen <felix@seita.nl> * Show spinner only while the promise is being fulfilled Signed-off-by: F.N. Claessen <felix@seita.nl> * Cancel previous request when the user makes a new request Signed-off-by: F.N. Claessen <felix@seita.nl> * Do not let spinner block the full page height, so the sensor table navigation can still be used Signed-off-by: F.N. Claessen <felix@seita.nl> * Change header and label colors inside the sidepanel to contrast against the sidepanel background Signed-off-by: F.N. Claessen <felix@seita.nl> * Style navbar logo to have a consistent height and adjust the width of the navbar-header accordingly Signed-off-by: F.N. Claessen <felix@seita.nl> * Actually load intended font Signed-off-by: F.N. Claessen <felix@seita.nl> * Enforce separation of time axis labels Signed-off-by: F.N. Claessen <felix@seita.nl> * Add return type annotation and docs: applying chart defaults returns a dictionary with vega-lite specs Signed-off-by: F.N. Claessen <felix@seita.nl> * Resolve hover glitch when exiting either the list of months or the list of years with the pointer. This stops the side panel from collapsing and reopening. Signed-off-by: F.N. Claessen <felix@seita.nl> * Enable swiping for left sidepanel Signed-off-by: F.N. Claessen <felix@seita.nl> * Stop using redundant litepicker plugin, which was messing with calendar styling Signed-off-by: F.N. Claessen <felix@seita.nl> * Fix test Signed-off-by: F.N. Claessen <felix@seita.nl> * Changelog entry Signed-off-by: F.N. Claessen <felix@seita.nl> * Synchronize styling of user pages Signed-off-by: F.N. Claessen <felix@seita.nl> * Style header action buttons Signed-off-by: F.N. Claessen <felix@seita.nl> * Fix test Signed-off-by: F.N. Claessen <felix@seita.nl> * Allow passing a default to get_attribute Signed-off-by: F.N. Claessen <felix@seita.nl> * Add timezone and timerange properties to asset Signed-off-by: F.N. Claessen <felix@seita.nl> * Expose asset properties pertaining to its sensor data as a new dev API endpoint Signed-off-by: F.N. Claessen <felix@seita.nl> * Speed up one case of belief searches Signed-off-by: F.N. Claessen <felix@seita.nl> * Move loading of js scripts to base.html Signed-off-by: F.N. Claessen <felix@seita.nl> * Add chart specs for showing multiple sensors Signed-off-by: F.N. Claessen <felix@seita.nl> * Add methods to search beliefs for an asset's sensors and for creating a chart of an asset's sensors Signed-off-by: F.N. Claessen <felix@seita.nl> * Add endpoints to retrieve an asset chart and chart data Signed-off-by: F.N. Claessen <felix@seita.nl> * Make sensor data a first class citizen of the asset page Signed-off-by: F.N. Claessen <felix@seita.nl> * Less margin around sensor listing on asset page Signed-off-by: F.N. Claessen <felix@seita.nl> * black and flake8 Signed-off-by: F.N. Claessen <felix@seita.nl> * Fix test Signed-off-by: F.N. Claessen <felix@seita.nl> * Use join instead of concat in order to handle non-unique join keys; that is, when sensor A has data for an event that sensor B does not have data for, the resulting DataFrame should get a row for that event, with a NaN value in the column of sensor B. Also, return an empty frame with the expected columns and indices in case no sensor was passed. Signed-off-by: F.N. Claessen <felix@seita.nl> * Refactor: avoid redundant join and filter in case account_name is None Signed-off-by: F.N. Claessen <felix@seita.nl> * Future annotations Signed-off-by: F.N. Claessen <felix@seita.nl> * Allow showing public sensors, too Signed-off-by: F.N. Claessen <felix@seita.nl> * Fix sidepanel text color for small screens Signed-off-by: F.N. Claessen <felix@seita.nl> * Add clickable sidepanel labels Signed-off-by: F.N. Claessen <felix@seita.nl> * Refactor sidepanel script to base Signed-off-by: F.N. Claessen <felix@seita.nl> * Keep all indices, instead of just those of the first sensor Signed-off-by: F.N. Claessen <felix@seita.nl> * Resample to the smallest resolution Signed-off-by: F.N. Claessen <felix@seita.nl> * Expand chart descriptions Signed-off-by: F.N. Claessen <felix@seita.nl> * Rename default asset chart Signed-off-by: F.N. Claessen <felix@seita.nl> * Increase default height of vertically concatenated charts Signed-off-by: F.N. Claessen <felix@seita.nl> * Show tooltip of the nearest data point when hovering over the chart Signed-off-by: F.N. Claessen <felix@seita.nl> * Remove unused transform Signed-off-by: F.N. Claessen <felix@seita.nl> * Add line layer Signed-off-by: F.N. Claessen <felix@seita.nl> * Sort sensors by the order given in sensor_ids, and avoid double sensor listings for public assets Signed-off-by: F.N. Claessen <felix@seita.nl> * Make sensors_to_show an asset property, and avoid getting redundant chart data Signed-off-by: F.N. Claessen <felix@seita.nl> * Refactor Signed-off-by: F.N. Claessen <felix@seita.nl> * Pass sensors to show if requesting a chart including data Signed-off-by: F.N. Claessen <felix@seita.nl> * Add timezone warning Signed-off-by: F.N. Claessen <felix@seita.nl> * Cannot compute minimum event resolution without data Signed-off-by: F.N. Claessen <felix@seita.nl> * Fix the with_appcontext_if_needed decorator in case `flexmeasures run` is used to set up a local development server Signed-off-by: F.N. Claessen <felix@seita.nl> * Only start spinner upon loading data, not upon loading the page or the chart. Signed-off-by: F.N. Claessen <felix@seita.nl> * Don't bother initializing the picker to specific dates in case of non-existent sensor data Signed-off-by: F.N. Claessen <felix@seita.nl> * Make timescale axis match the requested date range, by updating the chart specs upon picking a date Signed-off-by: F.N. Claessen <felix@seita.nl> * Use separate traces for each source, color by source name, and add source info to tooltip Signed-off-by: F.N. Claessen <felix@seita.nl> * Avoid requiring new dependency Signed-off-by: F.N. Claessen <felix@seita.nl> * Show previous results while waiting for new results Signed-off-by: F.N. Claessen <felix@seita.nl> * Fix calendar tooltip visibility Signed-off-by: F.N. Claessen <felix@seita.nl> * Custom date range styling in line with other button groups Signed-off-by: F.N. Claessen <felix@seita.nl> * Apply default legend font size also to layered views like vertically concatenated charts Signed-off-by: F.N. Claessen <felix@seita.nl> * Sync legend position on sensor and asset page, and allow longer labels Signed-off-by: F.N. Claessen <felix@seita.nl> * Fix merge_vega_lite_specs Signed-off-by: F.N. Claessen <felix@seita.nl> * Rename asset attribute Signed-off-by: F.N. Claessen <felix@seita.nl> * Changelog entries Signed-off-by: F.N. Claessen <felix@seita.nl> * Show tooltips using custom layer instead of voronoi Signed-off-by: F.N. Claessen <felix@seita.nl> * Use shared smallest resolution rather than original sensor resolution Signed-off-by: F.N. Claessen <felix@seita.nl> * Fix search_beliefs with significant speed-up Signed-off-by: F.N. Claessen <felix@seita.nl> * Fix missing variable Signed-off-by: F.N. Claessen <felix@seita.nl> * Workaround for timely-beliefs issue #104 Signed-off-by: F.N. Claessen <felix@seita.nl> * Slice previous results to avoid a poorly scaled x-axis when going from some time interval to a sub-interval. Signed-off-by: F.N. Claessen <felix@seita.nl> * Do not show custom layer for NaN data Signed-off-by: F.N. Claessen <felix@seita.nl> * black Signed-off-by: F.N. Claessen <felix@seita.nl> * Base timerange on sensors to show only Signed-off-by: F.N. Claessen <felix@seita.nl> * mypy Signed-off-by: F.N. Claessen <felix@seita.nl>
FlexMeasures · Jul 12, 2022 · 55589f3 · 55589f3
1 parent 6e134bd
commit 55589f3
Show file tree

Hide file tree

Showing 17 changed files with 993 additions and 225 deletions.
diff --git a/documentation/changelog.rst b/documentation/changelog.rst
@@ -7,6 +7,7 @@ v0.11.0 | June XX, 2022
 
 New features
 -------------
+* The asset page now shows the most relevant sensor data for the asset [see `PR #449 <http://www.github.com/FlexMeasures/flexmeasures/pull/449>`_]
 * Individual sensor charts show available annotations [see `PR #428 <http://www.github.com/FlexMeasures/flexmeasures/pull/428>`_]
 * Collapsible sidepanel (hover/swipe) used for date selection on sensor charts, and various styling improvements [see `PR #447 <http://www.github.com/FlexMeasures/flexmeasures/pull/447>`_ and `PR #448 <http://www.github.com/FlexMeasures/flexmeasures/pull/448>`_]
 * Switched from 12-hour AM/PM to 24-hour clock notation for time series chart axis labels [see `PR #446 <http://www.github.com/FlexMeasures/flexmeasures/pull/446>`_]
@@ -15,6 +16,7 @@ New features
 Bugfixes
 -----------
 * Do not fail asset page if entity addresses cannot be built [see `PR #457 <http://www.github.com/FlexMeasures/flexmeasures/pull/457>`_]
+* Time scale axes in sensor data charts now match the requested date range, rather than stopping at the edge of the available data [see `PR #449 <http://www.github.com/FlexMeasures/flexmeasures/pull/449>`_]
 
 Infrastructure / Support
 ----------------------

diff --git a/flexmeasures/api/dev/__init__.py b/flexmeasures/api/dev/__init__.py
@@ -5,7 +5,9 @@ def register_at(app: Flask):
     """This can be used to register FlaskViews."""
 
     from flexmeasures.api.dev.sensors import SensorAPI
+    from flexmeasures.api.dev.sensors import AssetAPI
 
     dev_api_prefix = "/api/dev"
 
     SensorAPI.register(app, route_prefix=dev_api_prefix)
+    AssetAPI.register(app, route_prefix=dev_api_prefix)
diff --git a/flexmeasures/api/dev/sensors.py b/flexmeasures/api/dev/sensors.py
@@ -9,8 +9,13 @@
 
 from flexmeasures.auth.policy import ADMIN_ROLE, ADMIN_READER_ROLE
 from flexmeasures.auth.decorators import permission_required_for_context
-from flexmeasures.data.schemas.sensors import SensorIdField
-from flexmeasures.data.schemas.times import AwareDateTimeField, DurationField
+from flexmeasures.data.schemas import (
+    AssetIdField,
+    AwareDateTimeField,
+    DurationField,
+    SensorIdField,
+)
+from flexmeasures.data.models.generic_assets import GenericAsset
 from flexmeasures.data.models.time_series import Sensor
 from flexmeasures.data.services.annotations import prepare_annotations_for_chart
 
@@ -130,6 +135,29 @@ def get(self, id: int, sensor: Sensor):
         return {attr: getattr(sensor, attr) for attr in attributes}
 
 
+class AssetAPI(FlaskView):
+    """
+    This view exposes asset attributes through API endpoints under development.
+    These endpoints are not yet part of our official API, but support the FlexMeasures UI.
+    """
+
+    route_base = "/asset"
+
+    @route("/<id>/")
+    @use_kwargs(
+        {"asset": AssetIdField(data_key="id")},
+        location="path",
+    )
+    @permission_required_for_context("read", arg_name="asset")
+    def get(self, id: int, asset: GenericAsset):
+        """GET from /asset/<id>
+
+        .. :quickref: Chart; Download asset attributes for use in charts
+        """
+        attributes = ["name", "timezone", "timerange_of_sensors_to_show"]
+        return {attr: getattr(asset, attr) for attr in attributes}
+
+
 def get_sensor_or_abort(id: int) -> Sensor:
     """
     Util function to help the GET requests. Will be obsolete..

diff --git a/flexmeasures/api/v3_0/assets.py b/flexmeasures/api/v3_0/assets.py
@@ -1,12 +1,16 @@
+import json
+
 from flask import current_app
 from flask_classful import FlaskView, route
 from flask_json import as_json
+from marshmallow import fields
 from webargs.flaskparser import use_kwargs, use_args
 
 from flexmeasures.auth.decorators import permission_required_for_context
 from flexmeasures.data import db
 from flexmeasures.data.models.user import Account
 from flexmeasures.data.models.generic_assets import GenericAsset
+from flexmeasures.data.schemas import AwareDateTimeField
 from flexmeasures.data.schemas.generic_assets import GenericAssetSchema as AssetSchema
 from flexmeasures.api.common.schemas.generic_assets import AssetIdField
 from flexmeasures.api.common.schemas.users import AccountIdField
@@ -231,3 +235,54 @@ def delete(self, id: int, asset: GenericAsset):
         db.session.commit()
         current_app.logger.info("Deleted asset '%s'." % asset_name)
         return {}, 204
+
+    @route("/<id>/chart/")
+    @use_kwargs(
+        {"asset": AssetIdField(data_key="id")},
+        location="path",
+    )
+    @use_kwargs(
+        {
+            "event_starts_after": AwareDateTimeField(format="iso", required=False),
+            "event_ends_before": AwareDateTimeField(format="iso", required=False),
+            "beliefs_after": AwareDateTimeField(format="iso", required=False),
+            "beliefs_before": AwareDateTimeField(format="iso", required=False),
+            "include_data": fields.Boolean(required=False),
+            "dataset_name": fields.Str(required=False),
+            "height": fields.Str(required=False),
+            "width": fields.Str(required=False),
+        },
+        location="query",
+    )
+    @permission_required_for_context("read", arg_name="asset")
+    def get_chart(self, id: int, asset: GenericAsset, **kwargs):
+        """GET from /assets/<id>/chart
+
+        .. :quickref: Chart; Download a chart with time series
+        """
+        return json.dumps(asset.chart(**kwargs))
+
+    @route("/<id>/chart_data/")
+    @use_kwargs(
+        {"asset": AssetIdField(data_key="id")},
+        location="path",
+    )
+    @use_kwargs(
+        {
+            "event_starts_after": AwareDateTimeField(format="iso", required=False),
+            "event_ends_before": AwareDateTimeField(format="iso", required=False),
+            "beliefs_after": AwareDateTimeField(format="iso", required=False),
+            "beliefs_before": AwareDateTimeField(format="iso", required=False),
+        },
+        location="query",
+    )
+    @permission_required_for_context("read", arg_name="asset")
+    def get_chart_data(self, id: int, asset: GenericAsset, **kwargs):
+        """GET from /assets/<id>/chart_data
+
+        .. :quickref: Chart; Download time series for use in charts
+
+        Data for use in charts (in case you have the chart specs already).
+        """
+        sensors = asset.sensors_to_show
+        return asset.search_beliefs(sensors=sensors, as_json=True, **kwargs)
diff --git a/flexmeasures/data/models/charts/belief_charts.py b/flexmeasures/data/models/charts/belief_charts.py
@@ -1,9 +1,15 @@
+from __future__ import annotations
+
+from datetime import datetime
+
 from flexmeasures.data.models.charts.defaults import FIELD_DEFINITIONS
 from flexmeasures.utils.flexmeasures_inflection import capitalize
 
 
 def bar_chart(
     sensor: "Sensor",  # noqa F821
+    event_starts_after: datetime | None = None,
+    event_ends_before: datetime | None = None,
     **override_chart_specs: dict,
 ):
     unit = sensor.unit if sensor.unit else "a.u."
@@ -14,32 +20,181 @@ def bar_chart(
         stack=None,
         **FIELD_DEFINITIONS["event_value"],
     )
+    event_start_field_definition = FIELD_DEFINITIONS["event_start"]
+    if event_starts_after and event_ends_before:
+        event_start_field_definition["scale"] = {
+            "domain": [
+                event_starts_after.timestamp() * 10**3,
+                event_ends_before.timestamp() * 10**3,
+            ]
+        }
+    resolution_in_ms = sensor.event_resolution.total_seconds() * 1000
     chart_specs = {
-        "description": "A simple bar chart.",
+        "description": "A simple bar chart showing sensor data.",
         "title": capitalize(sensor.name) if sensor.name != sensor.sensor_type else None,
-        "mark": "bar",
+        "mark": {
+            "type": "bar",
+            "clip": True,
+        },
         "encoding": {
-            "x": FIELD_DEFINITIONS["event_start"],
+            "x": event_start_field_definition,
             "x2": FIELD_DEFINITIONS["event_end"],
             "y": event_value_field_definition,
-            "color": FIELD_DEFINITIONS["source"],
+            "color": FIELD_DEFINITIONS["source_name"],
+            "detail": FIELD_DEFINITIONS["source"],
             "opacity": {"value": 0.7},
             "tooltip": [
                 FIELD_DEFINITIONS["full_date"],
                 {
                     **event_value_field_definition,
                     **dict(title=f"{capitalize(sensor.sensor_type)}"),
                 },
-                FIELD_DEFINITIONS["source"],
+                FIELD_DEFINITIONS["source_name"],
+                FIELD_DEFINITIONS["source_model"],
             ],
         },
         "transform": [
             {
-                "calculate": f"datum.event_start + {sensor.event_resolution.total_seconds() * 1000}",
+                "calculate": f"datum.event_start + {resolution_in_ms}",
                 "as": "event_end",
             },
         ],
     }
     for k, v in override_chart_specs.items():
         chart_specs[k] = v
     return chart_specs
+
+
+def chart_for_multiple_sensors(
+    sensors: list["Sensor"],  # noqa F821
+    event_starts_after: datetime | None = None,
+    event_ends_before: datetime | None = None,
+    **override_chart_specs: dict,
+):
+    sensors_specs = []
+    min_resolution_in_ms = (
+        min(sensor.event_resolution for sensor in sensors).total_seconds() * 1000
+    )
+    for sensor in sensors:
+        unit = sensor.unit if sensor.unit else "a.u."
+        event_value_field_definition = dict(
+            title=f"{capitalize(sensor.sensor_type)} ({unit})",
+            format=[".3s", unit],
+            formatType="quantityWithUnitFormat",
+            stack=None,
+            **{
+                **FIELD_DEFINITIONS["event_value"],
+                **dict(field=sensor.id),
+            },
+        )
+        event_start_field_definition = FIELD_DEFINITIONS["event_start"]
+        if event_starts_after and event_ends_before:
+            event_start_field_definition["scale"] = {
+                "domain": [
+                    event_starts_after.timestamp() * 10**3,
+                    event_ends_before.timestamp() * 10**3,
+                ]
+            }
+        shared_tooltip = [
+            FIELD_DEFINITIONS["full_date"],
+            {
+                **event_value_field_definition,
+                **dict(title=f"{capitalize(sensor.sensor_type)}"),
+            },
+            FIELD_DEFINITIONS["source_name"],
+            FIELD_DEFINITIONS["source_model"],
+        ]
+        sensor_specs = {
+            "title": capitalize(sensor.name)
+            if sensor.name != sensor.sensor_type
+            else None,
+            "layer": [
+                {
+                    "mark": {
+                        "type": "line",
+                        "interpolate": "step-after",
+                        "clip": True,
+                    },
+                    "encoding": {
+                        "x": event_start_field_definition,
+                        "y": event_value_field_definition,
+                        "color": FIELD_DEFINITIONS["source_name"],
+                        "detail": FIELD_DEFINITIONS["source"],
+                    },
+                },
+                {
+                    "mark": {
+                        "type": "rect",
+                        "y2": "height",
+                        "opacity": 0,
+                    },
+                    "encoding": {
+                        "x": event_start_field_definition,
+                        "x2": FIELD_DEFINITIONS["event_end"],
+                        "y": {
+                            "condition": {
+                                "test": "isNaN(datum['event_value'])",
+                                **event_value_field_definition,
+                            },
+                            "value": 0,
+                        },
+                        "detail": FIELD_DEFINITIONS["source"],
+                        "tooltip": shared_tooltip,
+                    },
+                    "transform": [
+                        {
+                            "calculate": f"datum.event_start + {min_resolution_in_ms}",
+                            "as": "event_end",
+                        },
+                    ],
+                },
+                {
+                    "mark": {
+                        "type": "circle",
+                        "opacity": 1,
+                        "clip": True,
+                    },
+                    "encoding": {
+                        "x": event_start_field_definition,
+                        "y": event_value_field_definition,
+                        "color": FIELD_DEFINITIONS["source_name"],
+                        "detail": FIELD_DEFINITIONS["source"],
+                        "size": {
+                            "condition": {
+                                "value": "200",
+                                "test": {"param": "paintbrush", "empty": False},
+                            },
+                            "value": "0",
+                        },
+                        "tooltip": shared_tooltip,
+                    },
+                    "params": [
+                        {
+                            "name": "paintbrush",
+                            "select": {
+                                "type": "point",
+                                "encodings": ["x"],
+                                "on": "mouseover",
+                                "nearest": False,
+                            },
+                        },
+                    ],
+                },
+            ],
+            "width": "container",
+        }
+        sensors_specs.append(sensor_specs)
+    chart_specs = dict(
+        description="A vertically concatenated chart showing sensor data.",
+        vconcat=[*sensors_specs],
+        spacing=100,
+        bounds="flush",
+    )
+    chart_specs["config"] = {
+        "view": {"continuousWidth": 800, "continuousHeight": 150},
+        "autosize": {"type": "fit-x", "contains": "padding"},
+    }
+    chart_specs["resolve"] = {"scale": {"x": "shared"}}
+    for k, v in override_chart_specs.items():
+        chart_specs[k] = v
+    return chart_specs