Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FLEXMEASURES_MAX_PLANNING_HORIZON can be int #583

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions documentation/changelog.rst
Expand Up @@ -10,6 +10,7 @@ v0.13.0 | February XX, 2023

New features
-------------
* The ``FLEXMEASURES_MAX_PLANNING_HORIZON`` config setting can also be set as an integer number of planning steps rather than just as a fixed duration, which makes it possible to schedule further ahead in coarser time steps [see `PR #583 <https://www.github.com/FlexMeasures/flexmeasures/pull/583>`_]

Bugfixes
-----------
Expand Down
7 changes: 4 additions & 3 deletions documentation/configuration.rst
Expand Up @@ -243,7 +243,7 @@ Default: ``timedelta(days=1)``
FLEXMEASURES_PLANNING_TTL
^^^^^^^^^^^^^^^^^^^^^^^^^

Time to live for UDI event ids of successful scheduling jobs. Set a negative timedelta to persist forever.
Time to live for schedule UUIDs of successful scheduling jobs. Set a negative timedelta to persist forever.

Default: ``timedelta(days=7)``

Expand All @@ -264,9 +264,10 @@ FLEXMEASURES_MAX_PLANNING_HORIZON

The maximum horizon for making schedules.
API users are not able to request longer schedules.
Set to ``None`` to forgo this limitation.
Can be set to a specific ``datetime.timedelta`` or to an integer number of planning steps, where the duration of a planning step is equal to the resolution of the applicable power sensor.
Set to ``None`` to forgo this limitation altoghether.

Default: ``timedelta(days=7, hours=1)``
Default: ``2520`` (e.g. 7 days for a 4-minute resolution sensor, 105 days for a 1-hour resolution sensor)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was just thinking about this, and it seems to me that this is a rather large default horizon. This will eat a lot of CPU budget.
To me, ~10% of this is reasonable (a bit over 2 weeks for a 1-hour sensor, a day for a 5-minute sensor)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also a thought for the future: this might have to change per account (at first via an attribute, I guess), as this is possibly relevant for billing flex scheduling services.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was just thinking about this, and it seems to me that this is a rather large default horizon. This will eat a lot of CPU budget.

This is not the default horizon (which is currently set to 2 days), but the default max horizon. I agree that this is a rather large value, but it is what the FlexMeasures scheduler can currently handle without the scheduling jobs timing out after 180 seconds.

I also thought of coupling this PR with the introduction of a config setting to allow FlexMeasures hosts to overwrite python-rq's default job timeout of 180 seconds for the scheduling queue, but I decided to save that for a separate PR. That would be a simple change in app.py, something like:

Queue(name="scheduling", default_timeout=app.config["FLEXMEASURES_PLANNING_TIMEOUT"])

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yes, sorry for the oversight. Actually, I would advise we give FLEXMEASURES_PLANNING_HORIZON the same treatment (that it can be an int).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought about that, too, but I didn't find the use case particularly convincing. Is it a worthwhile extra feature, or just resolves an itchy asymmetry?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The latter. But these things matter for user experience. I would expect these two to work the same.
And I believe we are not breaking existing settings here. Now is the cheapest time to do this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now is the cheapest time to do this.

I started implementing your request, but it turns out this is probably not true. The FLEXMEASURES_PLANNING_HORIZON is used in API versions 1.2 and 1.3, which allow retrieving schedules for multiple sensors in a single API call. That code assumes that the planning horizon is the same for all of these sensors, which with an integer FLEXMEASURES_PLANNING_HORIZON would no longer be the case. So this would currently require refactoring of already deprecated code.

I'd say it is cheaper to implement your request after these API versions are sunset.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds reasonable, can we make a ticket for version 0.14?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.



Access Tokens
Expand Down
2 changes: 1 addition & 1 deletion flexmeasures/api/v3_0/sensors.py
Expand Up @@ -245,7 +245,7 @@ def trigger_schedule( # noqa: C901
The length of the schedule can be set explicitly through the 'duration' field.
Otherwise, it is set by the config setting :ref:`planning_horizon_config`, which defaults to 48 hours.
If the flex-model contains targets that lie beyond the planning horizon, the length of the schedule is extended to accommodate them.
Finally, the schedule length is limited by :ref:`max_planning_horizon_config`, which defaults to 169 hours.
Finally, the schedule length is limited by :ref:`max_planning_horizon_config`, which defaults to 2520 steps of the sensor's resolution.
Targets that exceed the max planning horizon are not accepted.

The appropriate algorithm is chosen by FlexMeasures (based on asset type).
Expand Down
12 changes: 7 additions & 5 deletions flexmeasures/data/models/planning/storage.py
Expand Up @@ -19,6 +19,7 @@
)
from flexmeasures.data.schemas.scheduling.storage import StorageFlexModelSchema
from flexmeasures.data.schemas.scheduling import FlexContextSchema
from flexmeasures.utils.time_utils import get_max_planning_horizon


class StorageScheduler(Scheduler):
Expand Down Expand Up @@ -251,7 +252,9 @@ def deserialize_flex_config(self):
self.ensure_soc_min_max()

# Now it's time to check if our flex configurations holds up to schemas
self.flex_model = StorageFlexModelSchema(self.start).load(self.flex_model)
self.flex_model = StorageFlexModelSchema(
start=self.start, sensor=self.sensor
).load(self.flex_model)
self.flex_context = FlexContextSchema().load(self.flex_context)

# Extend schedule period in case a target exceeds its end
Expand All @@ -273,9 +276,7 @@ def possibly_extend_end(self):
[soc_target["datetime"] for soc_target in soc_targets]
)
if max_target_datetime > self.end:
max_server_horizon = current_app.config.get(
"FLEXMEASURES_MAX_PLANNING_HORIZON"
)
max_server_horizon = get_max_planning_horizon(self.resolution)
if max_server_horizon:
self.end = min(max_target_datetime, self.start + max_server_horizon)
else:
Expand Down Expand Up @@ -378,8 +379,9 @@ def build_device_soc_targets(
) # otherwise DST would be problematic
if target_datetime > end_of_schedule:
# Skip too-far-into-the-future target
max_server_horizon = get_max_planning_horizon(resolution)
current_app.logger.warning(
f'Disregarding target datetime {target_datetime}, because it exceeds {end_of_schedule}. Maximum scheduling horizon is {current_app.config.get("FLEXMEASURES_MAX_PLANNING_HORIZON")}.'
f"Disregarding target datetime {target_datetime}, because it exceeds {end_of_schedule}. Maximum scheduling horizon is {max_server_horizon}."
)
continue

Expand Down
17 changes: 10 additions & 7 deletions flexmeasures/data/schemas/scheduling/storage.py
Expand Up @@ -3,9 +3,10 @@
from datetime import datetime

from flask import current_app
from marshmallow import Schema, post_load, validate, validates, fields
from marshmallow import Schema, post_load, validate, validates_schema, fields
from marshmallow.validate import OneOf

from flexmeasures.data.models.time_series import Sensor
from flexmeasures.data.schemas.times import AwareDateTimeField
from flexmeasures.data.schemas.units import QuantityField
from flexmeasures.utils.unit_utils import ur
Expand Down Expand Up @@ -47,23 +48,25 @@ class StorageFlexModelSchema(Schema):
)
prefer_charging_sooner = fields.Bool(data_key="prefer-charging-sooner")

def __init__(self, start: datetime, *args, **kwargs):
def __init__(self, start: datetime, sensor: Sensor, *args, **kwargs):
"""Pass the schedule's start, so we can use it to validate soc-target datetimes."""
self.start = start
self.sensor = sensor
super().__init__(*args, **kwargs)

@validates("soc_targets")
def check_whether_targets_exceed_max_planning_horizon(
self, soc_targets: list[dict[str, datetime | float]]
):
@validates_schema
def check_whether_targets_exceed_max_planning_horizon(self, data: dict, **kwargs):
soc_targets: list[dict[str, datetime | float]] | None = data.get("soc_targets")
if not soc_targets:
return
max_server_horizon = current_app.config.get("FLEXMEASURES_MAX_PLANNING_HORIZON")
if isinstance(max_server_horizon, int):
max_server_horizon *= self.sensor.event_resolution
max_target_datetime = max([target["datetime"] for target in soc_targets])
max_server_datetime = self.start + max_server_horizon
if max_target_datetime > max_server_datetime:
current_app.logger.warning(
f'Target datetime exceeds {max_server_datetime}. Maximum scheduling horizon is {current_app.config.get("FLEXMEASURES_MAX_PLANNING_HORIZON")}.'
f"Target datetime exceeds {max_server_datetime}. Maximum scheduling horizon is {max_server_horizon}."
)

@post_load
Expand Down
2 changes: 1 addition & 1 deletion flexmeasures/utils/config_defaults.py
Expand Up @@ -115,7 +115,7 @@ class Config(object):
FLEXMEASURES_LP_SOLVER: str = "cbc"
FLEXMEASURES_JOB_TTL: timedelta = timedelta(days=1)
FLEXMEASURES_PLANNING_HORIZON: timedelta = timedelta(days=2)
FLEXMEASURES_MAX_PLANNING_HORIZON: timedelta | None = timedelta(days=7, hours=1)
FLEXMEASURES_MAX_PLANNING_HORIZON: timedelta | int | None = 2520 # smallest number divisible by 1-10, which yields pleasant-looking durations for common sensor resolutions
FLEXMEASURES_PLANNING_TTL: timedelta = timedelta(
days=7
) # Time to live for UDI event ids of successful scheduling jobs. Set a negative timedelta to persist forever.
Expand Down
9 changes: 9 additions & 0 deletions flexmeasures/utils/time_utils.py
Expand Up @@ -360,3 +360,12 @@ def to_http_time(dt: pd.Timestamp | datetime) -> str:
IMF-fixdate: https://www.rfc-editor.org/rfc/rfc7231#section-7.1.1.1
"""
return dt.strftime("%a, %d %b %Y %H:%M:%S GMT")


def get_max_planning_horizon(resolution: timedelta) -> timedelta | None:
"""Determine the maximum planning horizon for the given sensor resolution."""
max_planning_horizon = current_app.config.get("FLEXMEASURES_MAX_PLANNING_HORIZON")
if isinstance(max_planning_horizon, int):
# Config setting specifies maximum number of planning steps
max_planning_horizon *= resolution
return max_planning_horizon