Issue 427 chart option to include annotations (#428)

The /sensors/<id>/chart endpoint gets three new options to include annotations.
The /sensors/<id> endpoint shows asset annotations by default.
Also upgrades vega, vega-lite and vega-embed, and assigns charts a responsive width by default.


* Use SensorIdField

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Fix function call

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Layer annotations charts on top

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add API endpoint to fetch chart annotations

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Wrap long annotations

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Stack multiple annotations for the same event

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Fetch annotations in sensors.html

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Allow returning annotations as DataFrame

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Refactor into util function

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Fix import

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Allow including annotations in /chart endpoint

Signed-off-by: F.N. Claessen <felix@seita.nl>

* black

Signed-off-by: F.N. Claessen <felix@seita.nl>

* black

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Update vega dependencies

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Do not stack annotations if there are none

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Only add annotation chart specs if a dataset_name is given

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Visualize annotated data with shading

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Parameterize spacing

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Annotation font size and style

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add tooltip

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Increase tooltip font size

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Fix full_date transform

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Refactor: move layer definitions

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Don't cut off tooltip annotations at 7em (which is the default vega-tooltip css)

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Remove tooltip in favour of showing annotations only underneath the graph

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Optional inclusion of annotations in charts, and in sensor charts include sensor annotations and asset annotations by default

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Fix annotation stacking

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Include annotations that only fall partly within the queried time window

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Changelog entry

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Deprecate util function that is no longer used

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Revert "Deprecate util function that is no longer used"

This reverts commit 9bb9ffd.

* Fix individual sensor chart UI page

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Use auth decorator for read permissions on the sensor

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Fix default title font size for layered charts

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Wrap and stack annotations also in the UI endpoint for standalone sensor charts

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Clarify return type of sensor.chart()

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add missing type annotation

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add note from feedback on PR #428

Signed-off-by: F.N. Claessen <felix@seita.nl>

documentation/changelog.rst

@@ -7,6 +7,7 @@ v0.11.0 | June XX, 2022
 
 New features
 -------------
+* Individual sensor charts show available annotations [see `PR #428 <http://www.github.com/FlexMeasures/flexmeasures/pull/428>`_]
 
 Bugfixes
 -----------

flexmeasures/api/dev/sensors.py

@@ -1,50 +1,62 @@
 import json
+import warnings
 
 from flask_classful import FlaskView, route
-from flask_security import current_user, auth_required
+from flask_security import current_user
 from marshmallow import fields
 from webargs.flaskparser import use_kwargs
 from werkzeug.exceptions import abort
 
 from flexmeasures.auth.policy import ADMIN_ROLE, ADMIN_READER_ROLE
+from flexmeasures.auth.decorators import permission_required_for_context
+from flexmeasures.data.schemas.sensors import SensorIdField
 from flexmeasures.data.schemas.times import AwareDateTimeField
 from flexmeasures.data.models.time_series import Sensor
+from flexmeasures.data.services.annotations import prepare_annotations_for_chart
 
 
 class SensorAPI(FlaskView):
     """
     This view exposes sensor attributes through API endpoints under development.
     These endpoints are not yet part of our official API, but support the FlexMeasures UI.
-
-    TODO: use permission-based auth and marshmallow (SensorIDField).
     """
 
     route_base = "/sensor"
-    decorators = [auth_required()]
 
     @route("/<id>/chart/")
+    @use_kwargs(
+        {"sensor": SensorIdField(data_key="id")},
+        location="path",
+    )
     @use_kwargs(
         {
             "event_starts_after": AwareDateTimeField(format="iso", required=False),
             "event_ends_before": AwareDateTimeField(format="iso", required=False),
             "beliefs_after": AwareDateTimeField(format="iso", required=False),
             "beliefs_before": AwareDateTimeField(format="iso", required=False),
             "include_data": fields.Boolean(required=False),
+            "include_sensor_annotations": fields.Boolean(required=False),
+            "include_asset_annotations": fields.Boolean(required=False),
+            "include_account_annotations": fields.Boolean(required=False),
             "dataset_name": fields.Str(required=False),
             "height": fields.Str(required=False),
             "width": fields.Str(required=False),
         },
         location="query",
     )
-    def get_chart(self, id: int, **kwargs):
+    @permission_required_for_context("read", arg_name="sensor")
+    def get_chart(self, id: int, sensor: Sensor, **kwargs):
         """GET from /sensor/<id>/chart
 
         .. :quickref: Chart; Download a chart with time series
         """
-        sensor = get_sensor_or_abort(id)
         return json.dumps(sensor.chart(**kwargs))
 
     @route("/<id>/chart_data/")
+    @use_kwargs(
+        {"sensor": SensorIdField(data_key="id")},
+        location="path",
+    )
     @use_kwargs(
         {
             "event_starts_after": AwareDateTimeField(format="iso", required=False),
@@ -54,30 +66,77 @@ def get_chart(self, id: int, **kwargs):
         },
         location="query",
     )
-    def get_chart_data(self, id: int, **kwargs):
+    @permission_required_for_context("read", arg_name="sensor")
+    def get_chart_data(self, id: int, sensor: Sensor, **kwargs):
         """GET from /sensor/<id>/chart_data
 
         .. :quickref: Chart; Download time series for use in charts
 
         Data for use in charts (in case you have the chart specs already).
         """
-        sensor = get_sensor_or_abort(id)
         return sensor.search_beliefs(as_json=True, **kwargs)
 
-    def get(self, id: int):
+    @route("/<id>/chart_annotations/")
+    @use_kwargs(
+        {"sensor": SensorIdField(data_key="id")},
+        location="path",
+    )
+    @use_kwargs(
+        {
+            "event_starts_after": AwareDateTimeField(format="iso", required=False),
+            "event_ends_before": AwareDateTimeField(format="iso", required=False),
+            "beliefs_after": AwareDateTimeField(format="iso", required=False),
+            "beliefs_before": AwareDateTimeField(format="iso", required=False),
+        },
+        location="query",
+    )
+    @permission_required_for_context("read", arg_name="sensor")
+    def get_chart_annotations(self, id: int, sensor: Sensor, **kwargs):
+        """GET from /sensor/<id>/chart_annotations
+
+        .. :quickref: Chart; Download annotations for use in charts
+
+        Annotations for use in charts (in case you have the chart specs already).
+        """
+        event_starts_after = kwargs.get("event_starts_after", None)
+        event_ends_before = kwargs.get("event_ends_before", None)
+        df = sensor.generic_asset.search_annotations(
+            annotations_after=event_starts_after,
+            annotations_before=event_ends_before,
+            as_frame=True,
+        )
+
+        # Wrap and stack annotations
+        df = prepare_annotations_for_chart(df)
+
+        # Return JSON records
+        df = df.reset_index()
+        df["source"] = df["source"].astype(str)
+        return df.to_json(orient="records")
+
+    @route("/<id>/")
+    @use_kwargs(
+        {"sensor": SensorIdField(data_key="id")},
+        location="path",
+    )
+    @permission_required_for_context("read", arg_name="sensor")
+    def get(self, id: int, sensor: Sensor):
         """GET from /sensor/<id>
 
         .. :quickref: Chart; Download sensor attributes for use in charts
         """
-        sensor = get_sensor_or_abort(id)
         attributes = ["name", "timezone", "timerange"]
         return {attr: getattr(sensor, attr) for attr in attributes}
 
 
 def get_sensor_or_abort(id: int) -> Sensor:
     """
-    Util function to help the GET requests. Will be obsolete, see TODO above.
+    Util function to help the GET requests. Will be obsolete..
     """
+    warnings.warn(
+        "Util function will be deprecated. Switch to using SensorIdField to suppress this warning.",
+        FutureWarning,
+    )
     sensor = Sensor.query.filter(Sensor.id == id).one_or_none()
     if sensor is None:
         raise abort(404, f"Sensor {id} not found")

flexmeasures/data/models/annotations.py

@@ -201,3 +201,17 @@ def get_or_create_annotation(
     if annotation in db.session:
         db.session.expunge(annotation)
     return existing_annotation
+
+
+def to_annotation_frame(annotations: List[Annotation]) -> pd.DataFrame:
+    """Transform a list of annotations into a DataFrame.
+
+    We don't use a BeliefsDataFrame here, because they are designed for quantitative data only.
+    """
+    return pd.DataFrame(
+        [
+            [a.start, a.end, a.belief_time, a.source, a.type, a.content]
+            for a in annotations
+        ],
+        columns=["start", "end", "belief_time", "source", "type", "content"],
+    )

flexmeasures/data/models/charts/defaults.py

@@ -5,8 +5,9 @@
 
 
 FONT_SIZE = 16
+ANNOTATION_MARGIN = 16
 HEIGHT = 300
-WIDTH = 600
+WIDTH = "container"
 REDUCED_HEIGHT = REDUCED_WIDTH = 60
 SELECTOR_COLOR = "darkred"
 TIME_FORMAT = "%I:%M %p on %A %b %e, %Y"
@@ -37,14 +38,77 @@
         title="Time and date",
     ),
 }
+SHADE_LAYER = {
+    "mark": {
+        "type": "bar",
+        "color": "#bbbbbb",
+        "opacity": 0.3,
+        "size": HEIGHT,
+    },
+    "encoding": {
+        "x": dict(
+            field="start",
+            type="temporal",
+            title=None,
+        ),
+        "x2": dict(
+            field="end",
+            type="temporal",
+            title=None,
+        ),
+    },
+    "params": [
+        {
+            "name": "highlight",
+            "select": {"type": "point", "on": "mouseover"},
+        },
+        {"name": "select", "select": "point"},
+    ],
+}
+TEXT_LAYER = {
+    "mark": {
+        "type": "text",
+        "y": HEIGHT,
+        "dy": FONT_SIZE + ANNOTATION_MARGIN,
+        "baseline": "top",
+        "align": "left",
+        "fontSize": FONT_SIZE,
+        "fontStyle": "italic",
+    },
+    "encoding": {
+        "x": dict(
+            field="start",
+            type="temporal",
+            title=None,
+        ),
+        "text": {"type": "nominal", "field": "content"},
+        "opacity": {
+            "condition": [
+                {
+                    "param": "select",
+                    "empty": False,
+                    "value": 1,
+                },
+                {
+                    "param": "highlight",
+                    "empty": False,
+                    "value": 1,
+                },
+            ],
+            "value": 0,
+        },
+    },
+}
 LEGIBILITY_DEFAULTS = dict(
     config=dict(
         axis=dict(
             titleFontSize=FONT_SIZE,
             labelFontSize=FONT_SIZE,
-        )
+        ),
+        title=dict(
+            fontSize=FONT_SIZE,
+        ),
     ),
-    title=dict(fontSize=FONT_SIZE),
     encoding=dict(
         color=dict(
             dict(
@@ -66,6 +130,7 @@ def apply_chart_defaults(fn):
     @wraps(fn)
     def decorated_chart_specs(*args, **kwargs):
         dataset_name = kwargs.pop("dataset_name", None)
+        include_annotations = kwargs.pop("include_annotations", None)
         if isinstance(fn, Callable):
             # function that returns a chart specification
             chart_specs: Union[dict, alt.TopLevelMixin] = fn(*args, **kwargs)
@@ -75,8 +140,33 @@ def decorated_chart_specs(*args, **kwargs):
         if isinstance(chart_specs, alt.TopLevelMixin):
             chart_specs = chart_specs.to_dict()
             chart_specs.pop("$schema")
+
+        # Add transform function to calculate full date
+        if "transform" not in chart_specs:
+            chart_specs["transform"] = []
+        chart_specs["transform"].append(
+            {
+                "as": "full_date",
+                "calculate": f"timeFormat(datum.event_start, '{TIME_FORMAT}')",
+            }
+        )
+
         if dataset_name:
             chart_specs["data"] = {"name": dataset_name}
+            if include_annotations:
+                annotation_shades_layer = SHADE_LAYER
+                annotation_text_layer = TEXT_LAYER
+                annotation_shades_layer["data"] = {
+                    "name": dataset_name + "_annotations"
+                }
+                annotation_text_layer["data"] = {"name": dataset_name + "_annotations"}
+                chart_specs = {
+                    "layer": [
+                        annotation_shades_layer,
+                        chart_specs,
+                        annotation_text_layer,
+                    ]
+                }
 
         # Fall back to default height and width, if needed
         if "height" not in chart_specs:
@@ -90,15 +180,6 @@ def decorated_chart_specs(*args, **kwargs):
             chart_specs,
         )
 
-        # Add transform function to calculate full date
-        if "transform" not in chart_specs:
-            chart_specs["transform"] = []
-        chart_specs["transform"].append(
-            {
-                "as": "full_date",
-                "calculate": f"timeFormat(datum.event_start, '{TIME_FORMAT}')",
-            }
-        )
         return chart_specs
 
     return decorated_chart_specs