feat: add ProfitOrLossReporter to compute profit/loss of energy flo…

…ws (#808) * add CostReporter with tests Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * rename CostReporter to profit reporter Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * improve validation Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix comments Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * Add `is_profit` to `ProfitOrLossReporter` Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * Call a `Reporter` saved in the DB via CLI (#818) * add DataSource field to flexmeasures add report Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * add new command to create basic sources for the available reporters. Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * Switch off HiGHS logs for `LOGGING_LEVEL=INFO` (#824) * add log toggling for HiGHS Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * add changelog entry Signed-off-by: Victor Garcia Reolid <victor@seita.nl> --------- Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * rename command + add changelog Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * use for loop in flexmeasures add sources Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: typo Signed-off-by: F.N. Claessen <felix@seita.nl> * refactor: make 'kind' choice singular, so it matches the key in data_generators Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: add inline todo Signed-off-by: F.N. Claessen <felix@seita.nl> * refactor: get db from where we normally get it Signed-off-by: F.N. Claessen <felix@seita.nl> * style: black Signed-off-by: F.N. Claessen <felix@seita.nl> --------- Signed-off-by: Victor Garcia Reolid <victor@seita.nl> Signed-off-by: F.N. Claessen <felix@seita.nl> Co-authored-by: F.N. Claessen <felix@seita.nl> * Make source optional in the input field of the `AggregatorReporter` (#819) * add DataSource field to flexmeasures add report Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: use source and make it an optional field Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix tests Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * add test + fix case of multiple sources Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * add new command to create basic sources for the available reporters. Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * set sensor in ouput bdf Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * set resolution Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * update docstring Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * add changelog entry Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * rename command + add changelog Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * allow passing multiple sources Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * extend changelog Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * clarify docstring Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * check sources Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * use for loop in flexmeasures add sources Signed-off-by: Victor Garcia Reolid <victor@seita.nl> --------- Signed-off-by: Victor Garcia Reolid <victor@seita.nl> Signed-off-by: Victor <victor@seita.nl> * rename is_profit to profit_is_positive Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: `highs` string is in `solver_name` (#826) * fix: `highs` string is in `solve_name` using `in` Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * chore: update changelog Signed-off-by: Victor Garcia Reolid <victor@seita.nl> --------- Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * Fix: public sensors to show (#830) Fix two bugs dealing with public assets. * fix: public sensors have None owner Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: changelog entry Signed-off-by: F.N. Claessen <felix@seita.nl> * fix: searching for annotations on public assets Signed-off-by: F.N. Claessen <felix@seita.nl> * refactor: more explicit if-statement Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: add second fix to changelog Signed-off-by: F.N. Claessen <felix@seita.nl> --------- Signed-off-by: F.N. Claessen <felix@seita.nl> * add changelog entry Signed-off-by: Victor Garcia Reolid <victor@seita.nl> --------- Signed-off-by: Victor Garcia Reolid <victor@seita.nl> Signed-off-by: F.N. Claessen <felix@seita.nl> Signed-off-by: Victor <victor@seita.nl> Co-authored-by: F.N. Claessen <felix@seita.nl> Co-authored-by: Felix Claessen <30658763+Flix6x@users.noreply.github.com>
FlexMeasures · Aug 22, 2023 · a21059a · a21059a
1 parent 3c1ca31
commit a21059a
Show file tree

Hide file tree

Showing 19 changed files with 934 additions and 64 deletions.
diff --git a/documentation/changelog.rst b/documentation/changelog.rst
@@ -6,9 +6,16 @@ FlexMeasures Changelog
 v0.15.1 | August XX, 2023
 ============================
 
+New features
+-------------
+
+* Introduce new reporter to compute profit/loss due to electricity flows: `ProfitOrLossReporter` [see `PR #808 <https://github.com/FlexMeasures/flexmeasures/pull/808>`_]
+
+
 Bugfixes
 -----------
 
+* Use the `source` to filter beliefs in the `AggregatorReporter` and fix the case of having multiple sources [see `PR #819 <https://github.com/FlexMeasures/flexmeasures/pull/819>`_]
 * Disable HiGHS logs on the standard output when `LOGGING_LEVEL=INFO` [see `PR #824 <https://github.com/FlexMeasures/flexmeasures/pull/824>`_ and `PR #826 <https://github.com/FlexMeasures/flexmeasures/pull/826>`_]
 * Fix showing sensor data on the asset page of public assets, and searching for annotations on public assets [see `PR #830 <https://github.com/FlexMeasures/flexmeasures/pull/830>`_]
 

diff --git a/documentation/cli/change_log.rst b/documentation/cli/change_log.rst
@@ -4,6 +4,11 @@
 FlexMeasures CLI Changelog
 **********************
 
+since v0.16.0 | September XX, 2023
+=======================================
+* Add command ``flexmeasures add sources`` to add the base `DataSources` for the `DataGenerators`.
+
+
 since v0.15.0 | August 9, 2023
 =================================
 

diff --git a/flexmeasures/cli/data_add.py b/flexmeasures/cli/data_add.py
@@ -43,6 +43,7 @@
     Sensor,
     TimedBelief,
 )
+from flexmeasures.data.models.data_sources import DataSource
 from flexmeasures.data.models.validation_utils import (
     check_required_attributes,
     MissingAttributeException,
@@ -56,6 +57,7 @@
     SensorIdField,
     TimeIntervalField,
 )
+from flexmeasures.data.schemas.sources import DataSourceIdField
 from flexmeasures.data.schemas.times import TimeIntervalSchema
 from flexmeasures.data.schemas.scheduling.storage import EfficiencyField
 from flexmeasures.data.schemas.sensors import SensorSchema
@@ -84,6 +86,44 @@ def fm_add_data():
     """FlexMeasures: Add data."""
 
 
+@fm_add_data.command("sources")
+@click.option(
+    "--kind",
+    default=["reporter"],
+    type=click.Choice(["reporter", "scheduler", "forecaster"]),
+    multiple=True,
+    help="What kind of data generators to consider in the creation of the basic DataSources. Defaults to `reporter`.",
+)
+@with_appcontext
+def add_sources(kind: List[str]):
+    """Create data sources for the data generators found registered in the
+    application and the plugins. Currently, this command only registers the
+    sources for the Reporters.
+    """
+
+    for k in kind:
+        # todo: add other data-generators when adapted (and remove this check when all listed under our click.Choice are represented)
+        if k not in ("reporter",):
+            click.secho(f"Oh no, we don't support kind '{k}' yet.", **MsgStyle.WARN)
+            continue
+        click.echo(f"Adding `DataSources` for the {k} data generators.")
+
+        for name, data_generator in app.data_generators[k].items():
+            ds_info = data_generator.get_data_source_info()
+
+            # add empty data_generator configuration
+            ds_info["attributes"] = {"data_generator": {"config": {}, "parameters": {}}}
+
+            source = get_or_create_source(**ds_info)
+
+            click.secho(
+                f"Done. DataSource for data generator `{name}` is `{source}`.",
+                **MsgStyle.SUCCESS,
+            )
+
+    db.session.commit()
+
+
 @fm_add_data.command("account-role")
 @with_appcontext
 @click.option("--name", required=True)
@@ -1314,6 +1354,13 @@ def add_schedule_process(
     type=click.File("r"),
     help="Path to the JSON or YAML file with the configuration of the reporter.",
 )
+@click.option(
+    "--source",
+    "source",
+    required=False,
+    type=DataSourceIdField(),
+    help="DataSource ID of the `Reporter`.",
+)
 @click.option(
     "--parameters",
     "parameters_file",
@@ -1407,6 +1454,7 @@ def add_schedule_process(
 )
 def add_report(  # noqa: C901
     reporter_class: str,
+    source: DataSource | None = None,
     config_file: TextIOBase | None = None,
     parameters_file: TextIOBase | None = None,
     start: datetime | None = None,
@@ -1508,29 +1556,51 @@ def add_report(  # noqa: C901
 
     click.echo(f"Report scope:\n\tstart: {start}\n\tend:   {end}")
 
-    click.echo(
-        f"Looking for the Reporter {reporter_class} among all the registered reporters...",
-    )
+    if source is None:
 
-    # get reporter class
-    ReporterClass: Type[Reporter] = app.data_generators.get("reporter").get(
-        reporter_class
-    )
+        click.echo(
+            f"Looking for the Reporter {reporter_class} among all the registered reporters...",
+        )
 
-    # check if it exists
-    if ReporterClass is None:
-        click.secho(
-            f"Reporter class `{reporter_class}` not available.",
-            **MsgStyle.ERROR,
+        # get reporter class
+        ReporterClass: Type[Reporter] = app.data_generators.get("reporter").get(
+            reporter_class
         )
-        raise click.Abort()
 
-    click.secho(f"Reporter {reporter_class} found.", **MsgStyle.SUCCESS)
+        # check if it exists
+        if ReporterClass is None:
+            click.secho(
+                f"Reporter class `{reporter_class}` not available.",
+                **MsgStyle.ERROR,
+            )
+            raise click.Abort()
+
+        click.secho(f"Reporter {reporter_class} found.", **MsgStyle.SUCCESS)
 
-    # initialize reporter class with the reporter sensor and reporter config
-    reporter: Reporter = ReporterClass(config=config, save_config=save_config)
+        # initialize reporter class with the reporter sensor and reporter config
+        reporter: Reporter = ReporterClass(config=config, save_config=save_config)
 
-    click.echo("Report computation is running...")
+    else:
+        try:
+            reporter: Reporter = source.data_generator  # type: ignore
+
+            if not isinstance(reporter, Reporter):
+                raise NotImplementedError(
+                    f"DataGenerator `{reporter}` is not of the type `Reporter`"
+                )
+
+            click.secho(
+                f"Reporter `{reporter.__class__.__name__}` fetched successfully from the database.",
+                **MsgStyle.SUCCESS,
+            )
+
+        except NotImplementedError:
+            click.secho(
+                f"Error! DataSource `{source}` not storing a valid Reporter.",
+                **MsgStyle.ERROR,
+            )
+
+        reporter._save_config = save_config
 
     if ("start" not in parameters) and (start is not None):
         parameters["start"] = start.isoformat()
@@ -1539,6 +1609,8 @@ def add_report(  # noqa: C901
     if ("resolution" not in parameters) and (resolution is not None):
         parameters["resolution"] = pd.Timedelta(resolution).isoformat()
 
+    click.echo("Report computation is running...")
+
     # compute the report
     results: BeliefsDataFrame = reporter.compute(parameters=parameters)
 

diff --git a/flexmeasures/cli/tests/conftest.py b/flexmeasures/cli/tests/conftest.py
@@ -8,12 +8,15 @@
 from flexmeasures.data.models.time_series import Sensor, TimedBelief
 
 
-@pytest.fixture(scope="module")
+@pytest.fixture(scope="function")
 @pytest.mark.skip_github
-def setup_dummy_asset(db, app):
+def setup_dummy_asset(fresh_db, app):
     """
     Create an Asset to add sensors to and return the id.
     """
+
+    db = fresh_db
+
     dummy_asset_type = GenericAssetType(name="DummyGenericAssetType")
 
     db.session.add(dummy_asset_type)
@@ -27,14 +30,16 @@ def setup_dummy_asset(db, app):
     return dummy_asset.id
 
 
-@pytest.fixture(scope="module")
+@pytest.fixture(scope="function")
 @pytest.mark.skip_github
-def setup_dummy_data(db, app, setup_dummy_asset):
+def setup_dummy_data(fresh_db, app, setup_dummy_asset):
     """
     Create an asset with two sensors (1 and 2), and add the same set of 200 beliefs with an hourly resolution to each of them.
     Return the two sensors and a result sensor (which has no data).
     """
 
+    db = fresh_db
+
     report_asset_type = GenericAssetType(name="ReportAssetType")
 
     db.session.add(report_asset_type)
@@ -94,14 +99,19 @@ def setup_dummy_data(db, app, setup_dummy_asset):
 
 
 @pytest.mark.skip_github
-@pytest.fixture(scope="module")
-def process_power_sensor(db, app, add_market_prices):
+@pytest.fixture(scope="function")
+def process_power_sensor(
+    fresh_db,
+    app,
+):
     """
     Create an asset of type "process", power sensor to hold the result of
     the scheduler and price data consisting of 8 expensive hours, 8 cheap hours, and again 8 expensive hours-
 
     """
 
+    db = fresh_db
+
     process_asset_type = GenericAssetType(name="process")
 
     db.session.add(process_asset_type)

diff --git a/flexmeasures/cli/tests/test_data_add.py b/flexmeasures/cli/tests/test_data_add.py
@@ -98,7 +98,7 @@ def test_cli_help(app):
 
 
 @pytest.mark.skip_github
-def test_add_reporter(app, db, setup_dummy_data):
+def test_add_reporter(app, fresh_db, setup_dummy_data):
     """
     The reporter aggregates input data from two sensors (both have 200 data points)
     to a two-hour resolution.
@@ -154,6 +154,9 @@ def test_add_reporter(app, db, setup_dummy_data):
 
     cli_input = to_flags(cli_input_params)
 
+    # store config into config
+    cli_input.append("--save-config")
+
     # run test in an isolated file system
     with runner.isolated_filesystem():
 
@@ -200,9 +203,8 @@ def test_add_reporter(app, db, setup_dummy_data):
     previous_command_end = cli_input_params.get("end").replace(" ", "+")
 
     cli_input_params = {
-        "config": "reporter_config.json",
+        "source": stored_report.sources[0].id,
         "parameters": "parameters.json",
-        "reporter": "PandasReporter",
         "output-file": "test.csv",
         "timezone": "UTC",
     }
@@ -230,7 +232,10 @@ def test_add_reporter(app, db, setup_dummy_data):
             report_sensor_id
         )  # get fresh report sensor instance
 
-        assert "Reporter PandasReporter found" in result.output
+        assert (
+            "Reporter `PandasReporter` fetched successfully from the database."
+            in result.output
+        )
         assert f"Report computation done for sensor `{report_sensor}`." in result.output
 
         stored_report = report_sensor.search_beliefs(
@@ -242,7 +247,7 @@ def test_add_reporter(app, db, setup_dummy_data):
 
 
 @pytest.mark.skip_github
-def test_add_multiple_output(app, db, setup_dummy_data):
+def test_add_multiple_output(app, fresh_db, setup_dummy_data):
     """ """
 
     from flexmeasures.cli.data_add import add_report
@@ -337,7 +342,9 @@ def test_add_multiple_output(app, db, setup_dummy_data):
 
 @pytest.mark.skip_github
 @pytest.mark.parametrize("process_type", [("INFLEXIBLE"), ("SHIFTABLE"), ("BREAKABLE")])
-def test_add_process(app, process_power_sensor, process_type):
+def test_add_process(
+    app, process_power_sensor, process_type, add_market_prices_fresh_db
+):
     """
     Schedule a 4h of consumption block at a constant power of 400kW in a day using
     the three process policies: INFLEXIBLE, SHIFTABLE and BREAKABLE.

diff --git a/flexmeasures/conftest.py b/flexmeasures/conftest.py
@@ -528,6 +528,24 @@ def create_beliefs(db: SQLAlchemy, setup_markets, setup_sources) -> int:
 @pytest.fixture(scope="module")
 def add_market_prices(
     db: SQLAlchemy, setup_assets, setup_markets, setup_sources
+) -> dict[str, Sensor]:
+    return add_market_prices_common(db, setup_assets, setup_markets, setup_sources)
+
+
+@pytest.fixture(scope="function")
+def add_market_prices_fresh_db(
+    fresh_db: SQLAlchemy,
+    setup_assets_fresh_db,
+    setup_markets_fresh_db,
+    setup_sources_fresh_db,
+) -> dict[str, Sensor]:
+    return add_market_prices_common(
+        fresh_db, setup_assets_fresh_db, setup_markets_fresh_db, setup_sources_fresh_db
+    )
+
+
+def add_market_prices_common(
+    db: SQLAlchemy, setup_assets, setup_markets, setup_sources
 ) -> dict[str, Sensor]:
     """Add three days of market prices for the EPEX day-ahead market."""
 
@@ -607,7 +625,7 @@ def add_market_prices(
     ]
     db.session.add_all(day3_beliefs_production)
 
-    yield {
+    return {
         "epex_da": setup_markets["epex_da"],
         "epex_da_production": setup_markets["epex_da_production"],
     }

diff --git a/flexmeasures/data/models/data_sources.py b/flexmeasures/data/models/data_sources.py
@@ -254,7 +254,7 @@ def __init__(
         db.Model.__init__(self, **kwargs)
 
     @property
-    def data_generator(self):
+    def data_generator(self) -> DataGenerator:
         if self._data_generator:
             return self._data_generator