Add replay button to sensor and asset pages (#463)

Add a replay button that runs through the current date selection and updates the charts shown on the page (asset page or sensor page). This shows how new information is published at certain times and how forecasts and schedules are updated in real time accordingly. * Add replay button to sensor page Signed-off-by: F.N. Claessen <felix@seita.nl> * Add replay time below replay button Signed-off-by: F.N. Claessen <felix@seita.nl> * Include older beliefs in replay Signed-off-by: F.N. Claessen <felix@seita.nl> * Show only most recent beliefs in replay at any given time, and add new slice to simulatedData instead of reslicing result all over (also time function performance) Signed-off-by: F.N. Claessen <felix@seita.nl> * Speed up: no need to resize Signed-off-by: F.N. Claessen <felix@seita.nl> * Make sure vega view is embedded in page before attempting to change its dataset Signed-off-by: F.N. Claessen <felix@seita.nl> * Typo Signed-off-by: F.N. Claessen <felix@seita.nl> * Add replay button to asset page Instead of having multiple columns with event_values, named by sensor_id, we switched to one column containing the sensor_id, in addition to the column holding the event_values. Signed-off-by: F.N. Claessen <felix@seita.nl> * Pass along sensor information as an object Signed-off-by: F.N. Claessen <felix@seita.nl> * black Signed-off-by: F.N. Claessen <felix@seita.nl> * missing type annotations Signed-off-by: F.N. Claessen <felix@seita.nl> * Nicer js formatting Signed-off-by: F.N. Claessen <felix@seita.nl> * Remove obsolete code Signed-off-by: F.N. Claessen <felix@seita.nl> * Refactor: move partition function into replay-utils.js Signed-off-by: F.N. Claessen <felix@seita.nl> * Add dict representation of sensor to JSON representation of BeliefsDataFrame and refactor: move updateBeliefs function into replay-utils.js Signed-off-by: F.N. Claessen <felix@seita.nl> * Remove duplicate swipe functionality on the sensor page (already contained in leftsidepanel block) Signed-off-by: F.N. Claessen <felix@seita.nl> * Refactor: move sensor chart setup to base Signed-off-by: F.N. Claessen <felix@seita.nl> * Refactor: simplify Signed-off-by: F.N. Claessen <felix@seita.nl> * Refactor: rename and simplify Signed-off-by: F.N. Claessen <felix@seita.nl> * Add inline comments Signed-off-by: F.N. Claessen <felix@seita.nl> * Add docstrings to replay-utils.js Signed-off-by: F.N. Claessen <felix@seita.nl> * Rename parameter Signed-off-by: F.N. Claessen <felix@seita.nl> * Remove obsolete variable Signed-off-by: F.N. Claessen <felix@seita.nl> * Rename simulation/playback to replay Signed-off-by: F.N. Claessen <felix@seita.nl> * Rename function Signed-off-by: F.N. Claessen <felix@seita.nl> * Pass beliefs data rather than fetch result to replay function Signed-off-by: F.N. Claessen <felix@seita.nl> * Add docstring to replay function Signed-off-by: F.N. Claessen <felix@seita.nl> * Move beliefTimedelta to replay-utils.js Signed-off-by: F.N. Claessen <felix@seita.nl> * Remove logging statements Signed-off-by: F.N. Claessen <felix@seita.nl> * Changelog entry Signed-off-by: F.N. Claessen <felix@seita.nl> * Add missing semicolons Signed-off-by: F.N. Claessen <felix@seita.nl> * Add inline comments Signed-off-by: F.N. Claessen <felix@seita.nl> * Remove obsolete commented out lines Signed-off-by: F.N. Claessen <felix@seita.nl> * Expand docstring of partition function Signed-off-by: F.N. Claessen <felix@seita.nl> Signed-off-by: F.N. Claessen <felix@seita.nl> Signed-off-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com>
FlexMeasures · Sep 8, 2022 · d491371 · d491371
1 parent 51aa6aa
commit d491371
Show file tree

Hide file tree

Showing 12 changed files with 432 additions and 348 deletions.
diff --git a/documentation/changelog.rst b/documentation/changelog.rst
@@ -8,6 +8,7 @@ v0.12.0 | October XX, 2022
 New features
 -------------
 
+* Hit the replay button to replay what happened, available on the sensor and asset pages [see `PR #463 <http://www.github.com/FlexMeasures/flexmeasures/pull/463>`_]
 
 Bugfixes
 -----------

diff --git a/flexmeasures/api/dev/sensors.py b/flexmeasures/api/dev/sensors.py
@@ -71,6 +71,7 @@ def get_chart(self, id: int, sensor: Sensor, **kwargs):
             "beliefs_after": AwareDateTimeField(format="iso", required=False),
             "beliefs_before": AwareDateTimeField(format="iso", required=False),
             "resolution": DurationField(required=False),
+            "most_recent_beliefs_only": fields.Boolean(required=False),
         },
         location="query",
     )

diff --git a/flexmeasures/api/v3_0/assets.py b/flexmeasures/api/v3_0/assets.py
@@ -297,6 +297,7 @@ def get_chart(self, id: int, asset: GenericAsset, **kwargs):
             "event_ends_before": AwareDateTimeField(format="iso", required=False),
             "beliefs_after": AwareDateTimeField(format="iso", required=False),
             "beliefs_before": AwareDateTimeField(format="iso", required=False),
+            "most_recent_beliefs_only": fields.Boolean(required=False),
         },
         location="query",
     )

diff --git a/flexmeasures/data/models/charts/belief_charts.py b/flexmeasures/data/models/charts/belief_charts.py
@@ -87,10 +87,7 @@ def chart_for_multiple_sensors(
             format=[".3~r", unit],
             formatType="quantityWithUnitFormat",
             stack=None,
-            **{
-                **FIELD_DEFINITIONS["event_value"],
-                **dict(field=sensor.id),
-            },
+            **FIELD_DEFINITIONS["event_value"],
         )
         event_start_field_definition = FIELD_DEFINITIONS["event_start"]
         if event_starts_after and event_ends_before:
@@ -113,6 +110,7 @@ def chart_for_multiple_sensors(
             "title": capitalize(sensor.name)
             if sensor.name != sensor.sensor_type
             else None,
+            "transform": [{"filter": f"datum.sensor.id == {sensor.id}"}],
             "layer": [
                 {
                     "mark": {

diff --git a/flexmeasures/data/models/data_sources.py b/flexmeasures/data/models/data_sources.py
@@ -81,10 +81,10 @@ def description(self):
                 descr += f" v{self.version}"
         return descr
 
-    def __repr__(self):
+    def __repr__(self) -> str:
         return "<Data source %r (%s)>" % (self.id, self.description)
 
-    def __str__(self):
+    def __str__(self) -> str:
         return self.description
 
     def to_dict(self) -> dict:

diff --git a/flexmeasures/data/models/generic_assets.py b/flexmeasures/data/models/generic_assets.py
@@ -342,6 +342,7 @@ def search_beliefs(
         source: Optional[
             Union[DataSource, List[DataSource], int, List[int], str, List[str]]
         ] = None,
+        most_recent_beliefs_only: bool = True,
         most_recent_events_only: bool = False,
         as_json: bool = False,
     ) -> Union[BeliefsDataFrame, str]:
@@ -373,7 +374,7 @@ def search_beliefs(
                 horizons_at_least=horizons_at_least,
                 horizons_at_most=horizons_at_most,
                 source=source,
-                most_recent_beliefs_only=True,
+                most_recent_beliefs_only=most_recent_beliefs_only,
                 most_recent_events_only=most_recent_events_only,
                 one_deterministic_belief_per_event_per_source=True,
             )
@@ -394,18 +395,35 @@ def search_beliefs(
                     if bdf.event_resolution > timedelta(0):
                         bdf = bdf.resample_events(minimum_non_zero_resolution)
                     df = simplify_index(
-                        bdf, index_levels_to_columns=["source"]
-                    ).set_index(["source"], append=True)
-                    df_dict[sensor.id] = df.rename(columns=dict(event_value=sensor.id))
-                df = list(df_dict.values())[0].join(
-                    list(df_dict.values())[1:], how="outer"
-                )
+                        bdf,
+                        index_levels_to_columns=["source"]
+                        if most_recent_beliefs_only
+                        else ["belief_time", "source"],
+                    ).set_index(
+                        ["source"]
+                        if most_recent_beliefs_only
+                        else ["belief_time", "source"],
+                        append=True,
+                    )
+                    df["sensor"] = sensor  # or some JSONfiable representation
+                    df = df.set_index(["sensor"], append=True)
+                    df_dict[sensor.id] = df
+                df = pd.concat(df_dict.values())
             else:
                 df = simplify_index(
-                    BeliefsDataFrame(), index_levels_to_columns=["source"]
-                ).set_index(["source"], append=True)
+                    BeliefsDataFrame(),
+                    index_levels_to_columns=["source"]
+                    if most_recent_beliefs_only
+                    else ["belief_time", "source"],
+                ).set_index(
+                    ["source"]
+                    if most_recent_beliefs_only
+                    else ["belief_time", "source"],
+                    append=True,
+                )
             df = df.reset_index()
             df["source"] = df["source"].apply(lambda x: x.to_dict())
+            df["sensor"] = df["sensor"].apply(lambda x: x.to_dict())
             return df.to_json(orient="records")
         return bdf_dict
 

diff --git a/flexmeasures/data/models/time_series.py b/flexmeasures/data/models/time_series.py
@@ -339,6 +339,8 @@ def search_beliefs(
         )
         if as_json:
             df = bdf.reset_index()
+            df["sensor"] = self
+            df["sensor"] = df["sensor"].apply(lambda x: x.to_dict())
             df["source"] = df["source"].apply(lambda x: x.to_dict())
             return df.to_json(orient="records")
         return bdf
@@ -480,6 +482,15 @@ def timerange(self) -> Dict[str, datetime_type]:
     def __repr__(self) -> str:
         return f"<Sensor {self.id}: {self.name}, unit: {self.unit} res.: {self.event_resolution}>"
 
+    def __str__(self) -> str:
+        return self.name
+
+    def to_dict(self) -> dict:
+        return dict(
+            id=self.id,
+            name=self.name,
+        )
+
     @classmethod
     def find_closest(
         cls, generic_asset_type_name: str, sensor_name: str, n: int = 1, **kwargs

diff --git a/flexmeasures/ui/static/css/flexmeasures.css b/flexmeasures/ui/static/css/flexmeasures.css
@@ -1770,4 +1770,53 @@ div.heading-group {
     align-items: center;
 }
 
-/* ---- End Unsure / potential Legacy ---- */
+/* ---- End Unsure / potential Legacy ---- */
+
+
+/* ---- Play/Pause button ---- */
+.replay-container {
+  display: flex;
+  flex-direction: column;
+  align-items: center;
+  justify-content: center;
+}
+#replay-time {
+  margin-top: 15px;
+}
+#replay {
+  cursor: pointer !important;
+  margin-top: 190px;
+  width: 50px;
+  height: 50px;
+  position: relative;
+}
+#replay:before, #replay:after {
+  content: "";
+  position: absolute;
+  width: 0;
+  border-style: solid;
+  border-color: transparent;
+  border-left-color: var(--secondary-color);
+  transition: 0.3s;
+}
+#replay:not(.playing):before {
+  height: 50px;
+  border-width: 25px 0px 25px 50px;
+}
+#replay:not(.playing):after {
+  left: 25px;
+  top: 12.5px;
+  height: 0;
+  border-width: 12.5px 0 12.5px 25px;
+}
+#replay:before {
+  left: 0;
+  height: 50px;
+  border-width: 0 0 0 16.6666666667px;
+}
+#replay:after {
+  left: 33.3333333333px;
+  top: 0px;
+  height: 50px;
+  border-width: 0 0 0 16.6666666667px;
+}
diff --git a/flexmeasures/ui/static/js/replay-utils.js b/flexmeasures/ui/static/js/replay-utils.js
@@ -0,0 +1,44 @@
+// Replay utils
+
+/**
+ * Partitions array into two arrays.
+ *
+ * Partitions array into two array by pushing elements left or right given some decision function, which is
+ * evaluated on each element. Successful validations lead to placement on the left side, others on the right.
+ *
+ * @param {Array} array               Array to be partitioned.
+ * @param {function} decisionFunction Function that assigns elements to the left or right arrays.
+ * @return {Array}                    Array containing the left and right arrays.
+ */
+export function partition(array, decisionFunction){
+  return array.reduce(function(result, element, i) {
+    decisionFunction(element, i, array)
+      ? result[0].push(element)
+      : result[1].push(element);
+      return result;
+    }, [[],[]]
+  );
+};
+
+/**
+ * Updates beliefs.
+ *
+ * Updates oldBeliefs with the most recent newBeliefs about the same event for the same sensor by the same source.
+ *
+ * @param {Array} oldBeliefs Array containing old beliefs.
+ * @param {Array} newBeliefs Array containing new beliefs.
+ * @return {Array}           Array containing updated beliefs.
+ */
+export function updateBeliefs(oldBeliefs, newBeliefs) {
+  // Group by sensor, event start and source
+  var oldBeliefsByEventBySource = Object.fromEntries(new Map(oldBeliefs.map(belief => [belief.sensor.id + '_' + belief.event_start + '_' + belief.source.id, belief])));  // array -> dict (already had one belief per event)
+
+  // Group by sensor, event start and source, and select only the most recent new beliefs
+  var mostRecentNewBeliefsByEventBySource = Object.fromEntries(new Map(newBeliefs.map(belief => [belief.sensor.id + '_' + belief.event_start + '_' + belief.source.id, belief])));  // array -> dict (assumes beliefs are ordered by ascending belief time, with the last belief used as dict value)
+
+  // Return old beliefs updated with most recent new beliefs
+  return Object.values({...oldBeliefsByEventBySource, ...mostRecentNewBeliefsByEventBySource})  // dict -> array
+}
+
+//  Define the step duration for the replay (value in ms)
+export var beliefTimedelta = 3600000