feat: save/fetch Reporter as DataSource (#751)

* feat: revision to add `attributes` column to the `data_source` table Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: add `attributes` column to the DataSource model Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: add sensors relationship in DataSource Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: make sensors relationship viewonly Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: add report_config to Reporter class Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: add PandasReporter report and reporter schemas Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: update fixture by removing beliefs_search_configs and adding input_variables Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: add report config to PandasReporter Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: add helper methods to DataSource Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: modernize AggregatorReporter Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: add attributes hash Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: add attributes to the function get_or_create_source Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: add attribute hash to get_or_create_source Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: save/fetch data generator to/from data source Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * refactor: adapt reporters to use new DataGenerator class Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: use default method on load Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: adapt tests of the schemas Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: use a DataGenerator with a schema defined Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * changing backref from "dynamic" to "select" Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: add hash_attributes static method Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: use hash_attributes static method Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: adding attributes_hash to the DataSource unique constraint list Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: add constraint to migration and downgrade Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: only returning keys from the attributes field Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * refactor: rename _inputs_schema to _input_schema Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: typing Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: avoid future data leakage Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * refactor: rename PandasReporterInputConfigSchema to PandasReporterInputSchema Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * docs: clarify description of the fake_data mock variable Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * docs: fix docstring Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: use default value Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: allow creating new attributes with the method `set_attributes` Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * docs: add changelog entry Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * docs: fix docstring Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: use default value Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: allow creating new attributes with the method `set_attributes` Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * docs: add changelog entry Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: add reporters and schedulers into the data_generators attribute in the app context Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: raise Exceptions instead of returning None Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: move sensor attribute from config to inputs Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: use same structure for data generators and add test Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * refactor: use input_resolution instead of resolution Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * doc: update schema docstring Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * refactor: rename input_resolution to resolution Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: remove sensor from config Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * docs: add comment Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: remove df_output Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix:. for data in data["data"] haha Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * doc: add docstring to compute and __init__ in DataGenerator Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * refactor: rename inputs to input Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: removing constructor Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * docs: improve docstring Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * test: add data to confest Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * test: add test_dst_transition Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: never returning None Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * test: add test to check timely-beliefs resampling and calling an aggregator between to dates with different offsets, in this case, daylight saving transitions. Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * test: change output sensor id Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * docs: add docstring to the data_source propert of the class DataGenerator. Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * docs: edit data_source docstring Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * refactor: ranming input_sensors to input_variables Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * rename input to parameters Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * remove unnecessary import Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * add save_config attribute Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * remove leftover comment Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * add inline comments Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * deprecation message for app.reoprters and app.schedulers Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix typo Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * use data source created by the data generator Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: support YAML in `flexmeasures add report` command (#752) * feat: add pyyaml to the requirements Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: support YAML and add report_config Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: move `types-PyYAML` dependency to the right place Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: use a DataGenerator with defined schemas Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: adapt tests of the schemas Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: add option to open default editor Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: move sensor to input Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: parse resolution properly Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: remove accidentally commited file Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: avoid potential bug Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * rename input to parameters Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * add chagelog entry Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * add pyyaml to app.txt Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * add --save-config to the add_report command Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * improve changelog files Signed-off-by: Victor Garcia Reolid <victor@seita.nl> --------- Signed-off-by: Victor Garcia Reolid <victor@seita.nl> Signed-off-by: Victor <victor@seita.nl> * Polish `PandasReporter` schemas (#788) * add required_input and required_output to PandasReporterConfigSchema and input & outupt to parameters Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * adapt tests Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * implement multiple output and simplify tibber reporter Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix example in the docstring Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * remove max=1 constraint Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * add example for _clean_parameters Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * remove time parameters in input (_clean_parameters method) Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * remove filed added accidentally Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * improve assert Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * update changelog entry Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix changelog Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * Adapt `test_add_report` to use the new field of the `PandasReporter` schema (#789) * fix: typo Signed-off-by: F.N. Claessen <felix@seita.nl> * fix: fetch output sensor only from parameters dict Signed-off-by: F.N. Claessen <felix@seita.nl> * adapt the CLI to deal with multiple output Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix typos Signed-off-by: Victor Garcia Reolid <victor@seita.nl> --------- Signed-off-by: F.N. Claessen <felix@seita.nl> Signed-off-by: Victor Garcia Reolid <victor@seita.nl> Co-authored-by: F.N. Claessen <felix@seita.nl> --------- Signed-off-by: Victor Garcia Reolid <victor@seita.nl> Signed-off-by: F.N. Claessen <felix@seita.nl> Co-authored-by: F.N. Claessen <felix@seita.nl> --------- Signed-off-by: Victor Garcia Reolid <victor@seita.nl> Signed-off-by: Victor <victor@seita.nl> Signed-off-by: F.N. Claessen <felix@seita.nl> Co-authored-by: F.N. Claessen <felix@seita.nl>
FlexMeasures · Aug 7, 2023 · 2866237 · 2866237
1 parent b1a85e8
commit 2866237
Show file tree

Hide file tree

Showing 26 changed files with 1,507 additions and 649 deletions.
diff --git a/ci/run_mypy.sh b/ci/run_mypy.sh
@@ -1,7 +1,7 @@
 #!/bin/bash
 set -e
 pip install --upgrade 'mypy>=0.902'
-pip install types-pytz types-requests types-Flask types-click types-redis types-tzlocal types-python-dateutil types-setuptools types-tabulate
+pip install types-pytz types-requests types-Flask types-click types-redis types-tzlocal types-python-dateutil types-setuptools types-tabulate types-PyYAML
 # We are checking python files which have type hints, and leave out bigger issues we made issues for
 # * data/scripts: We'll remove legacy code: https://trello.com/c/1wEnHOkK/7-remove-custom-data-scripts
 # * data/models and data/services: https://trello.com/c/rGxZ9h2H/540-makequery-call-signature-is-incoherent

diff --git a/documentation/changelog.rst b/documentation/changelog.rst
@@ -25,6 +25,8 @@ New features
 * Added API endpoints `/sensors/<id>` for fetching a single sensor, `/sensors` (POST) for adding a sensor, `/sensors/<id>` (PATCH) for updating a sensor and `/sensors/<id>` (DELETE) for deleting a sensor. [see `PR #759 <https://www.github.com/FlexMeasures/flexmeasures/pull/759>`_] and [see `PR #767 <https://www.github.com/FlexMeasures/flexmeasures/pull/767>`_] and [see `PR #773 <https://www.github.com/FlexMeasures/flexmeasures/pull/773>`_] and [see `PR #784 <https://www.github.com/FlexMeasures/flexmeasures/pull/784>`_]
 * The CLI now allows to set lists and dicts as asset & sensor attributes (formerly only single values) [see `PR #762 <https://www.github.com/FlexMeasures/flexmeasures/pull/762>`_]
 * Add `ProcessScheduler` class to optimize the starting time of processes one of the policies developed (INFLEXIBLE, SHIFTABLE and BREAKABLE), accessible via the CLI command `flexmeasures add schedule for-process` [see `PR #729 <https://www.github.com/FlexMeasures/flexmeasures/pull/729>`_ and `PR #768 <https://www.github.com/FlexMeasures/flexmeasures/pull/768>`_]
+* Users will be able to see (e.g. in the UI) exactly which reporter created the report (saved as sensor data), and hosts will be able to identify exactly which configuration was used to create a given report [see `PR #751 <https://www.github.com/FlexMeasures/flexmeasures/pull/751>`_, `PR #752 <https://www.github.com/FlexMeasures/flexmeasures/pull/752>`_ and `PR #788 <https://www.github.com/FlexMeasures/flexmeasures/pull/788>`_]
+* The CLI `flexmeasures add report` now allows passing `config` and `parameters` in YAML format as files or editable via the system's default editor [see `PR #788 <https://www.github.com/FlexMeasures/flexmeasures/pull/788>`_]
 
 Bugfixes
 -----------

diff --git a/documentation/cli/change_log.rst b/documentation/cli/change_log.rst
@@ -9,6 +9,7 @@ since v0.15.0 | July XX, 2023
 
 * Allow deleting multiple sensors with a single call to ``flexmeasures delete sensor`` by passing the ``--id`` option multiple times.
 * Add ``flexmeasures add schedule for-process`` to create a new process schedule for a given power sensor.
+* Add support for describing ``config`` and ``parameters`` in YAML for the command ``flexmeasures add report``, editable in user's code editor using the flags ``--edit-config`` or ``--edit-parameters``.
 * Add ``--kind process`` option to create the asset and sensors for the ``ProcessScheduler`` tutorial.
 
 since v0.14.1 | June XX, 2023

diff --git a/flexmeasures/app.py b/flexmeasures/app.py
@@ -5,6 +5,7 @@
 from __future__ import annotations
 
 import time
+from copy import copy
 import os
 from pathlib import Path
 from datetime import date
@@ -123,8 +124,26 @@ def create(  # noqa C901
     from flexmeasures.utils.coding_utils import get_classes_module
     from flexmeasures.data.models import reporting, planning
 
-    app.reporters = get_classes_module("flexmeasures.data.models", reporting.Reporter)
-    app.schedulers = get_classes_module("flexmeasures.data.models", planning.Scheduler)
+    reporters = get_classes_module("flexmeasures.data.models", reporting.Reporter)
+    schedulers = get_classes_module("flexmeasures.data.models", planning.Scheduler)
+
+    app.data_generators = dict()
+    app.data_generators["reporter"] = copy(
+        reporters
+    )  # use copy to avoid mutating app.reporters
+    app.data_generators["scheduler"] = schedulers
+
+    # deprecation of app.reporters
+    app.reporters = reporters
+    app.schedulers = schedulers
+
+    def get_reporters():
+        app.logger.warning(
+            '`app.reporters` is deprecated. Use `app.data_generators["reporter"]` instead.'
+        )
+        return app.data_generators["reporter"]
+
+    setattr(app, "reporters", property(get_reporters))
 
     # add auth policy