Stop trimming the planning window in response to price availability (#…

…538)

Stop trimming the planning window in response to price availability, which is a problem when SoC targets occur outside of the available price window, by making a simplistic assumption about future (and past) prices.


* Prevent scheduler from trimming the planning window, using naive forecasting (forward filling the last known price); switching to naive forecasting with a seasonal periodicity of 1 day or 1 week would give inconsistent results, because they will then depend on how long a perice period has been loaded in the price bdf.

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Log a warning in case of extending prices to the edges of the planning window

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Update docstring with description of new behaviour

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Changelog entry

Signed-off-by: F.N. Claessen <felix@seita.nl>

Signed-off-by: F.N. Claessen <felix@seita.nl>

documentation/changelog.rst

@@ -21,6 +21,7 @@ Bugfixes
 -----------
 * The CLI command ``flexmeasures show beliefs`` now supports plotting time series data that includes NaN values, and provides better support for plotting multiple sensors that do not share the same unit [see `PR #516 <http://www.github.com/FlexMeasures/flexmeasures/pull/516>`_ and `PR #539 <http://www.github.com/FlexMeasures/flexmeasures/pull/539>`_]
 * Consistent CLI/UI support for asset lat/lng positions up to 7 decimal places (previously the UI rounded to 4 decimal places, whereas the CLI allowed more than 4) [see `PR #522 <http://www.github.com/FlexMeasures/flexmeasures/pull/522>`_]
+* Stop trimming the planning window in response to price availability, which is a problem when SoC targets occur outside of the available price window, by making a simplistic assumption about future prices [see `PR #538 <http://www.github.com/FlexMeasures/flexmeasures/pull/538>`_]
 * Faster loading of initial charts and calendar date selection [see `PR #533 <http://www.github.com/FlexMeasures/flexmeasures/pull/533>`_]
 
 Infrastructure / Support

flexmeasures/data/models/planning/storage.py

@@ -55,15 +55,15 @@ def schedule(
             beliefs_before=belief_time,
             price_sensor=consumption_price_sensor,
             sensor=sensor,
-            allow_trimmed_query_window=True,
+            allow_trimmed_query_window=False,
         )
         down_deviation_prices, (start, end) = get_prices(
             (start, end),
             resolution,
             beliefs_before=belief_time,
             price_sensor=production_price_sensor,
             sensor=sensor,
-            allow_trimmed_query_window=True,
+            allow_trimmed_query_window=False,
         )
 
         start = pd.Timestamp(start).tz_convert("UTC")

flexmeasures/data/models/planning/utils.py

@@ -163,7 +163,13 @@ def get_prices(
     sensor: Optional[Sensor] = None,
     allow_trimmed_query_window: bool = True,
 ) -> Tuple[pd.DataFrame, Tuple[datetime, datetime]]:
-    """Check for known prices or price forecasts, trimming query window accordingly if allowed."""
+    """Check for known prices or price forecasts.
+
+    If so allowed, the query window is trimmed according to the available data.
+    If not allowed, prices are extended to the edges of the query window:
+    - The first available price serves as a naive backcast.
+    - The last available price serves as a naive forecast.
+    """
 
     # Look for the applicable price sensor
     if price_sensor is None:
@@ -199,13 +205,29 @@ def get_prices(
             first_event_start = price_df.first_valid_index()
             last_event_end = price_df.last_valid_index() + resolution
             current_app.logger.warning(
-                f"Prices partially unknown for planning window (sensor {price_sensor.id}). Trimming planning window (from {query_window[0]} until {query_window[-1]}) to {first_event_start} until {last_event_end}."
+                f"Prices partially unknown for planning window (sensor {price_sensor.id}). "
+                f"Trimming planning window (from {query_window[0]} until {query_window[-1]}) to {first_event_start} until {last_event_end}."
             )
             query_window = (first_event_start, last_event_end)
         else:
-            raise UnknownPricesException(
-                f"Prices partially unknown for planning window (sensor {price_sensor.id})."
+            current_app.logger.warning(
+                f"Prices partially unknown for planning window (sensor {price_sensor.id}). "
+                f"Assuming the first price is valid from the start of the planning window ({query_window[0]}), "
+                f"and the last price is valid until the end of the planning window ({query_window[-1]})."
+            )
+            index = initialize_index(
+                start=query_window[0],
+                end=query_window[1],
+                resolution=resolution,
             )
+            price_df = price_df.reindex(index)
+            # or to also forward fill intermediate NaN values, use: price_df = price_df.ffill().bfill()
+            price_df[: price_df.first_valid_index()] = price_df[
+                price_df.index == price_df.first_valid_index()
+            ].values[0]
+            price_df[price_df.last_valid_index() :] = price_df[
+                price_df.index == price_df.last_valid_index()
+            ].values[0]
     return price_df, query_window