From 3c95759f169e517dbaa10087a1fc13cd6151226e Mon Sep 17 00:00:00 2001 From: Felix Claessen <30658763+Flix6x@users.noreply.github.com> Date: Tue, 22 Nov 2022 00:08:09 +0100 Subject: [PATCH] Issue 482 add scheduling test for maximizing self consumption (#532) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add scheduling test for maximizing self-consumption, and improve time series db queries for fixed tariffs (and other long-term constants). * Better documentation of flexibility model for storage in endpoint; refactor its parameters and handling within the code for readability Signed-off-by: Nicolas Höning * add changelog entry Signed-off-by: Nicolas Höning * make tests work, include updating older API versions, make prefer_charging_sooner part of storage specs & an optional parameter in API v3 Signed-off-by: Nicolas Höning * use storage_specs in CLI command, as well Signed-off-by: Nicolas Höning * remove default resolution of 15M, for now pass in what you want Signed-off-by: Nicolas Höning * various review comments Signed-off-by: Nicolas Höning * black Signed-off-by: Nicolas Höning * fix tests Signed-off-by: Nicolas Höning * always load sensor when checking storage specs Signed-off-by: Nicolas Höning * begin to handle source model and version during scheduling Signed-off-by: Nicolas Höning * we can get multiple sources from our query (in the old setting, when we use name, but also in the new setting, unless we always include the user_id) Signed-off-by: Nicolas Höning * give our two in-built schedulers an official __author__ and __version__ Signed-off-by: Nicolas Höning * review comments Signed-off-by: Nicolas Höning * refactor getting data source for a job to util function; use the actual data source ID for this lookup if possible Signed-off-by: Nicolas Höning * pass sensor to check_storage_specs, as we always have it already Signed-off-by: Nicolas Höning * wrap Scheduler in classes, unify data source handling a bit more Signed-off-by: Nicolas Höning * Support pandas 1.4 (#525) Add a pandas version check in initialize_index. * Use initialize_series util function Signed-off-by: F.N. Claessen * Update initialize_index for pandas>=1.4 Signed-off-by: F.N. Claessen * flake8 Signed-off-by: F.N. Claessen * Use initialize_index or initialize_series in all places where the closed keyword argument was used Signed-off-by: F.N. Claessen * flake8 Signed-off-by: F.N. Claessen * mypy: PEP 484 prohibits implicit Optional Signed-off-by: F.N. Claessen * black after mypy Signed-off-by: F.N. Claessen Signed-off-by: F.N. Claessen * Stop requiring min/max SoC attributes, which have defaults: - Default min = 0 - Default max = the highest target value, or np.nan if there are no targets, which subsequently maps to infinity in our solver Signed-off-by: F.N. Claessen * Set up device constraint columns for efficiencies in Charge Point scheduler, too Signed-off-by: F.N. Claessen * Derive flow constraints for battery scheduling, too (copied from Charge Point scheduler) Signed-off-by: F.N. Claessen * Refactor: rename BatteryScheduler to StorageScheduler Signed-off-by: F.N. Claessen * Warn for deprecation of schedule_battery and schedule_charging_station Signed-off-by: F.N. Claessen * Use StorageScheduler instead of ChargingStationScheduler Signed-off-by: F.N. Claessen * Deprecate ChargingStationScheduler Signed-off-by: F.N. Claessen * Refactor: move StorageScheduler to dedicated module Signed-off-by: F.N. Claessen * Update docstring Signed-off-by: F.N. Claessen * fix test Signed-off-by: F.N. Claessen * flake8 Signed-off-by: F.N. Claessen * Lose the v in version strings; prefer versions showing up as 'version: 3' over 'version: v3'. Even though Scheduler versioning does not necessarily need to follow semantic versioning (see discussion here: https://github.com/semver/semver/issues/235), the v is still redundant. Signed-off-by: F.N. Claessen * Refactor: rename module Signed-off-by: F.N. Claessen * Deal with empty SoC targets Signed-off-by: F.N. Claessen * Stop wrapping DataFrame representations in logging Signed-off-by: F.N. Claessen * Log warning instead of raising UnknownForecastException, and assume zero power values for missing values Signed-off-by: F.N. Claessen * Workaround for GH #484 Signed-off-by: F.N. Claessen * Test maximizing self-consumption Signed-off-by: F.N. Claessen * Refactor: a single parameterized test instead of 2 tests with a lot of duplicate code Signed-off-by: F.N. Claessen * Remove debug statements Signed-off-by: F.N. Claessen * black Signed-off-by: F.N. Claessen * Correct mistake while refactoring Signed-off-by: F.N. Claessen * Check default price sensor in both scenarios Signed-off-by: F.N. Claessen * Expand explanation of optimization context Signed-off-by: F.N. Claessen * Reorder assertions and add more comments Signed-off-by: F.N. Claessen * Upgrade timely-beliefs to partly resolve workaround Signed-off-by: F.N. Claessen * changelog entry Signed-off-by: F.N. Claessen Signed-off-by: Nicolas Höning Signed-off-by: F.N. Claessen Co-authored-by: Nicolas Höning --- documentation/changelog.rst | 1 + .../data/models/planning/tests/conftest.py | 56 +++++++++++++++ .../data/models/planning/tests/test_solver.py | 71 +++++++++++++++---- flexmeasures/data/models/time_series.py | 8 ++- requirements/app.in | 2 +- requirements/app.txt | 2 +- 6 files changed, 122 insertions(+), 18 deletions(-) diff --git a/documentation/changelog.rst b/documentation/changelog.rst index 3261f28a0..8c8dcbaa8 100644 --- a/documentation/changelog.rst +++ b/documentation/changelog.rst @@ -32,6 +32,7 @@ Infrastructure / Support * Plugins can save BeliefsSeries, too, instead of just BeliefsDataFrames [see `PR #523 `_] * Improve documentation and code w.r.t. storage flexibility modelling ― prepare for handling other schedulers & merge battery and car charging schedulers [see `PR #511 `_] * Revised strategy for removing unchanged beliefs when saving data: retain the oldest measurement (ex-post belief), too [see `PR #518 `_] +* Scheduling test for maximizing self-consumption, and improved time series db queries for fixed tariffs (and other long-term constants) [see `PR #532 `_] v0.11.3 | November 2, 2022 diff --git a/flexmeasures/data/models/planning/tests/conftest.py b/flexmeasures/data/models/planning/tests/conftest.py index e59db120f..acf8ca4ab 100644 --- a/flexmeasures/data/models/planning/tests/conftest.py +++ b/flexmeasures/data/models/planning/tests/conftest.py @@ -3,6 +3,7 @@ from datetime import timedelta import pytest +from timely_beliefs.sensors.func_store.knowledge_horizons import at_date import pandas as pd from flexmeasures.data.models.generic_assets import GenericAsset, GenericAssetType @@ -18,6 +19,61 @@ def setup_planning_test_data(db, add_market_prices, add_charging_station_assets) print("Setting up data for planning tests on %s" % db.engine) +@pytest.fixture(scope="module") +def create_test_tariffs(db, setup_accounts, setup_sources) -> dict[str, Sensor]: + """Create a fixed consumption tariff and a fixed feed-in tariff that is lower.""" + + market_type = GenericAssetType( + name="tariff market", + ) + db.session.add(market_type) + contract = GenericAsset( + name="supply contract", + generic_asset_type=market_type, + owner=setup_accounts["Supplier"], + ) + db.session.add(contract) + consumption_price_sensor = Sensor( + name="fixed consumption tariff", + generic_asset=contract, + event_resolution=timedelta(hours=24 * 365), + unit="EUR/MWh", + knowledge_horizon=(at_date, {"knowledge_time": "2014-11-01T00:00+01:00"}), + ) + db.session.add(consumption_price_sensor) + production_price_sensor = Sensor( + name="fixed feed-in tariff", + generic_asset=contract, + event_resolution=timedelta(hours=24 * 365), + unit="EUR/MWh", + knowledge_horizon=(at_date, {"knowledge_time": "2014-11-01T00:00+01:00"}), + ) + db.session.add(production_price_sensor) + + # Add prices + consumption_price = TimedBelief( + event_start="2015-01-01T00:00+01:00", + belief_time="2014-11-01T00:00+01:00", # publication date + event_value=300 * 1.21, + source=setup_sources["Seita"], + sensor=consumption_price_sensor, + ) + db.session.add(consumption_price) + production_price = TimedBelief( + event_start="2015-01-01T00:00+01:00", + belief_time="2014-11-01T00:00+01:00", # publication date + event_value=300, + source=setup_sources["Seita"], + sensor=production_price_sensor, + ) + db.session.add(production_price) + db.session.flush() # make sure that prices are assigned to price sensors + return { + "consumption_price_sensor": consumption_price_sensor, + "production_price_sensor": production_price_sensor, + } + + @pytest.fixture(scope="module") def building(db, setup_accounts, setup_markets) -> GenericAsset: """ diff --git a/flexmeasures/data/models/planning/tests/test_solver.py b/flexmeasures/data/models/planning/tests/test_solver.py index ef2b9711f..08e15334c 100644 --- a/flexmeasures/data/models/planning/tests/test_solver.py +++ b/flexmeasures/data/models/planning/tests/test_solver.py @@ -272,25 +272,52 @@ def test_fallback_to_unsolvable_problem(target_soc, charging_station_name): ) +@pytest.mark.parametrize( + "market_scenario", + [ + "dynamic contract", + "fixed contract", + ], +) def test_building_solver_day_2( db, add_battery_assets, add_market_prices, + create_test_tariffs, add_inflexible_device_forecasts, inflexible_devices, flexible_devices, + market_scenario: str, ): - """Check battery scheduling results within the context of a building with PV, for day 2, - which is set up with 8 expensive, then 8 cheap, then again 8 expensive hours. - We expect the scheduler to: - - completely discharge within the first 8 hours - - completely charge within the next 8 hours - - completely discharge within the last 8 hours + """Check battery scheduling results within the context of a building with PV, for day 2, against the following market scenarios: + 1) a dynamic tariff with equal consumption and feed-in tariffs, that is set up with 8 expensive, then 8 cheap, then again 8 expensive hours. + 2) a fixed consumption tariff and a fixed feed-in tariff that is lower, which incentives to maximize self-consumption of PV power into the battery. + In the test data: + - Hours with net production coincide with low dynamic market prices. + - Hours with net consumption coincide with high dynamic market prices. + So when the prices are low (in scenario 1), we have net production, and when they are high, net consumption. + That means we have first net consumption, then net production, and then net consumption again. + In either scenario, we expect the scheduler to: + - completely discharge within the first 8 hours (either due to 1) high prices, or 2) net consumption) + - completely charge within the next 8 hours (either due to 1) low prices, or 2) net production) + - completely discharge within the last 8 hours (either due to 1) high prices, or 2) net consumption) """ - epex_da = Sensor.query.filter(Sensor.name == "epex_da").one_or_none() battery = flexible_devices["battery power sensor"] building = battery.generic_asset - assert battery.get_attribute("market_id") == epex_da.id + default_consumption_price_sensor = Sensor.query.filter( + Sensor.name == "epex_da" + ).one_or_none() + assert battery.get_attribute("market_id") == default_consumption_price_sensor.id + if market_scenario == "dynamic contract": + consumption_price_sensor = default_consumption_price_sensor + production_price_sensor = consumption_price_sensor + elif market_scenario == "fixed contract": + consumption_price_sensor = create_test_tariffs["consumption_price_sensor"] + production_price_sensor = create_test_tariffs["production_price_sensor"] + else: + raise NotImplementedError( + f"Missing test case for market conditions '{market_scenario}'" + ) tz = pytz.timezone("Europe/Amsterdam") start = tz.localize(datetime(2015, 1, 2)) end = tz.localize(datetime(2015, 1, 3)) @@ -315,6 +342,8 @@ def test_building_solver_day_2( end, resolution, storage_specs=storage_specs, + consumption_price_sensor=consumption_price_sensor, + production_price_sensor=production_price_sensor, inflexible_device_sensors=inflexible_devices.values(), ) soc_schedule = integrate_time_series(schedule, soc_at_start, decimal_precision=6) @@ -353,14 +382,28 @@ def test_building_solver_day_2( assert soc >= max(soc_min, battery.get_attribute("min_soc_in_mwh")) assert soc <= battery.get_attribute("max_soc_in_mwh") - # Check whether the resulting soc schedule follows our expectations for 8 expensive, 8 cheap and 8 expensive hours - assert soc_schedule.iloc[-1] == max( - soc_min, battery.get_attribute("min_soc_in_mwh") - ) # Battery sold out at the end of its planning horizon + # Check whether the resulting soc schedule follows our expectations for. + # To recap, in scenario 1 and 2, the schedule should mainly be influenced by: + # 1) 8 expensive, 8 cheap and 8 expensive hours + # 2) 8 net-consumption, 8 net-production and 8 net-consumption hours + # Result after 8 hours + # 1) Sell what you begin with + # 2) The battery discharged as far as it could during the first 8 net-consumption hours assert soc_schedule.loc[start + timedelta(hours=8)] == max( soc_min, battery.get_attribute("min_soc_in_mwh") - ) # Sell what you begin with + ) + + # Result after second 8 hour-interval + # 1) Buy what you can to sell later, when prices will be high again + # 2) The battery charged with PV power as far as it could during the middle 8 net-production hours assert soc_schedule.loc[start + timedelta(hours=16)] == min( soc_max, battery.get_attribute("max_soc_in_mwh") - ) # Buy what you can to sell later + ) + + # Result at end of day + # 1) The battery sold out at the end of its planning horizon + # 2) The battery discharged as far as it could during the last 8 net-consumption hours + assert soc_schedule.iloc[-1] == max( + soc_min, battery.get_attribute("min_soc_in_mwh") + ) diff --git a/flexmeasures/data/models/time_series.py b/flexmeasures/data/models/time_series.py index 958e61201..d1acf04f6 100644 --- a/flexmeasures/data/models/time_series.py +++ b/flexmeasures/data/models/time_series.py @@ -652,8 +652,9 @@ def search( bdf = cls.search_session( session=db.session, sensor=sensor, - event_starts_after=event_starts_after, - event_ends_before=event_ends_before, + # Workaround (1st half) for https://github.com/FlexMeasures/flexmeasures/issues/484 + event_ends_after=event_starts_after, + event_starts_before=event_ends_before, beliefs_after=beliefs_after, beliefs_before=beliefs_before, horizons_at_least=horizons_at_least, @@ -690,6 +691,9 @@ def search( bdf = bdf.resample_events( resolution, keep_only_most_recent_belief=most_recent_beliefs_only ) + # Workaround (2nd half) for https://github.com/FlexMeasures/flexmeasures/issues/484 + bdf = bdf[bdf.event_starts >= event_starts_after] + bdf = bdf[bdf.event_ends <= event_ends_before] bdf_dict[bdf.sensor.name] = bdf if sum_multiple: diff --git a/requirements/app.in b/requirements/app.in index 82e15f9ec..df6a9a220 100644 --- a/requirements/app.in +++ b/requirements/app.in @@ -28,7 +28,7 @@ tldextract pyomo>=5.6 tabulate timetomodel>=0.7.1 -timely-beliefs[forecast]>=1.13 +timely-beliefs[forecast]>=1.14 python-dotenv # a backport, not needed in Python3.8 importlib_metadata diff --git a/requirements/app.txt b/requirements/app.txt index 71df651f4..480874aa5 100644 --- a/requirements/app.txt +++ b/requirements/app.txt @@ -314,7 +314,7 @@ tabulate==0.8.10 # via -r requirements/app.in threadpoolctl==3.1.0 # via scikit-learn -timely-beliefs[forecast]==1.13.0 +timely-beliefs[forecast]==1.14.0 # via -r requirements/app.in timetomodel==0.7.1 # via -r requirements/app.in