feat: support YAML in `flexmeasures add report` command #752

victorgarcia98 · 2023-06-28T10:51:36Z

This PR adapts the command flexmeasures add report to work with the new implementation of the Reporter, subclassing from DataGenerator.

Rename the CLI parameter --reporter-config to --config.
Introduce CLI parameter --inputs.
Introduce flags --edit-config and --edit-inputs to open the default editor to edit config and inputs. Similar work in feat: add flexmeasures add schedule generic command #730
Support YAML to describe config and inputs files, keeping the support for JSON.

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

…ure/reporting/add-report-support-yml

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

…ure/reporting/add-report-support-yml

…ure/reporting/add-report-support-yml Signed-off-by: Victor <victor@seita.nl>

flexmeasures/cli/data_add.py

Flix6x · 2023-07-02T13:45:13Z

flexmeasures/cli/data_add.py

+    if ("end" not in inputs) and (end is not None):
+        inputs["end"] = end.isoformat()
+    if ("resolution" not in inputs) and (resolution is not None):
+        inputs["resolution"] = pd.Timedelta.isoformat(resolution)


I wonder how this behaves for calendar days. Can we add a test for this? We don't need to solve this case immediately, but I'd like to know where we stand.

I wonder if we should just test pd.Timedelta.isoformat(resolution) in a DST changing day or a end to end test is needed. What do you think?

Let's go with the former, so just the function.

…ure/reporting/add-report-support-yml Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

…port-yml' into feature/reporting/add-report-support-yml Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

Flix6x

Thanks for the revisions.

I noted one accidental file, one potential bug and a request for additional tests. Let's add the changelog entry, too.

Untitled.ipynb

Flix6x · 2023-07-06T11:42:46Z

flexmeasures/cli/data_add.py

+    if ("end" not in inputs) and (end is not None):
+        inputs["end"] = end.isoformat()
+    if ("resolution" not in inputs) and (resolution is not None):
+        inputs["resolution"] = pd.Timedelta.isoformat(resolution)


Let's go with the former, so just the function.

flexmeasures/cli/data_add.py

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

Flix6x · 2023-08-01T19:43:01Z

Does this still require an update with regards to the renaming of input to parameters, or do you tackle that in another PR?

Flix6x · 2023-08-01T19:44:08Z

Otherwise, I believe with a changelog entry (main changelog and CLI changelog) this is close to being complete.

…ure/reporting/add-report-support-yml Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

victorgarcia98 · 2023-08-02T20:00:05Z

Thanks!

The changelog entry includes a reference to PR #751, as well.

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

…ure/reporting/add-report-support-yml Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

…ure/reporting/add-report-support-yml

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

Flix6x

Just add the missing changelog entry, and move the current one to #751.

documentation/changelog.rst

documentation/cli/change_log.rst

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

…ure/reporting/add-report-support-yml Signed-off-by: Victor <victor@seita.nl>

* feat: revision to add `attributes` column to the `data_source` table Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: add `attributes` column to the DataSource model Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: add sensors relationship in DataSource Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: make sensors relationship viewonly Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: add report_config to Reporter class Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: add PandasReporter report and reporter schemas Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: update fixture by removing beliefs_search_configs and adding input_variables Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: add report config to PandasReporter Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: add helper methods to DataSource Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: modernize AggregatorReporter Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: add attributes hash Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: add attributes to the function get_or_create_source Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: add attribute hash to get_or_create_source Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: save/fetch data generator to/from data source Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * refactor: adapt reporters to use new DataGenerator class Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: use default method on load Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: adapt tests of the schemas Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: use a DataGenerator with a schema defined Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * changing backref from "dynamic" to "select" Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: add hash_attributes static method Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: use hash_attributes static method Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: adding attributes_hash to the DataSource unique constraint list Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: add constraint to migration and downgrade Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: only returning keys from the attributes field Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * refactor: rename _inputs_schema to _input_schema Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: typing Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: avoid future data leakage Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * refactor: rename PandasReporterInputConfigSchema to PandasReporterInputSchema Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * docs: clarify description of the fake_data mock variable Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * docs: fix docstring Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: use default value Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: allow creating new attributes with the method `set_attributes` Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * docs: add changelog entry Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * docs: fix docstring Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: use default value Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: allow creating new attributes with the method `set_attributes` Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * docs: add changelog entry Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: add reporters and schedulers into the data_generators attribute in the app context Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: raise Exceptions instead of returning None Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: move sensor attribute from config to inputs Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: use same structure for data generators and add test Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * refactor: use input_resolution instead of resolution Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * doc: update schema docstring Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * refactor: rename input_resolution to resolution Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: remove sensor from config Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * docs: add comment Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: remove df_output Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix:. for data in data["data"] haha Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * doc: add docstring to compute and __init__ in DataGenerator Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * refactor: rename inputs to input Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: removing constructor Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * docs: improve docstring Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * test: add data to confest Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * test: add test_dst_transition Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: never returning None Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * test: add test to check timely-beliefs resampling and calling an aggregator between to dates with different offsets, in this case, daylight saving transitions. Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * test: change output sensor id Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * docs: add docstring to the data_source propert of the class DataGenerator. Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * docs: edit data_source docstring Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * refactor: ranming input_sensors to input_variables Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * rename input to parameters Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * remove unnecessary import Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * add save_config attribute Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * remove leftover comment Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * add inline comments Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * deprecation message for app.reoprters and app.schedulers Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix typo Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * use data source created by the data generator Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: support YAML in `flexmeasures add report` command (#752) * feat: add pyyaml to the requirements Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: support YAML and add report_config Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: move `types-PyYAML` dependency to the right place Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: use a DataGenerator with defined schemas Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: adapt tests of the schemas Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * feat: add option to open default editor Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: move sensor to input Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: parse resolution properly Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: remove accidentally commited file Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix: avoid potential bug Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * rename input to parameters Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * add chagelog entry Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * add pyyaml to app.txt Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * add --save-config to the add_report command Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * improve changelog files Signed-off-by: Victor Garcia Reolid <victor@seita.nl> --------- Signed-off-by: Victor Garcia Reolid <victor@seita.nl> Signed-off-by: Victor <victor@seita.nl> * Polish `PandasReporter` schemas (#788) * add required_input and required_output to PandasReporterConfigSchema and input & outupt to parameters Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * adapt tests Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * implement multiple output and simplify tibber reporter Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix example in the docstring Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * remove max=1 constraint Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * add example for _clean_parameters Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * remove time parameters in input (_clean_parameters method) Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * remove filed added accidentally Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * improve assert Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * update changelog entry Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix changelog Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * Adapt `test_add_report` to use the new field of the `PandasReporter` schema (#789) * fix: typo Signed-off-by: F.N. Claessen <felix@seita.nl> * fix: fetch output sensor only from parameters dict Signed-off-by: F.N. Claessen <felix@seita.nl> * adapt the CLI to deal with multiple output Signed-off-by: Victor Garcia Reolid <victor@seita.nl> * fix typos Signed-off-by: Victor Garcia Reolid <victor@seita.nl> --------- Signed-off-by: F.N. Claessen <felix@seita.nl> Signed-off-by: Victor Garcia Reolid <victor@seita.nl> Co-authored-by: F.N. Claessen <felix@seita.nl> --------- Signed-off-by: Victor Garcia Reolid <victor@seita.nl> Signed-off-by: F.N. Claessen <felix@seita.nl> Co-authored-by: F.N. Claessen <felix@seita.nl> --------- Signed-off-by: Victor Garcia Reolid <victor@seita.nl> Signed-off-by: Victor <victor@seita.nl> Signed-off-by: F.N. Claessen <felix@seita.nl> Co-authored-by: F.N. Claessen <felix@seita.nl>

victorgarcia98 added 8 commits June 23, 2023 16:03

feat: add pyyaml to the requirements

f968b09

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

feat: support YAML and add report_config

e0384c5

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

fix: move types-PyYAML dependency to the right place

86ed86c

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

Merge branch 'feature/reporting/save-reporters-data-source' into feat…

3bb577e

…ure/reporting/add-report-support-yml

fix: use a DataGenerator with defined schemas

632534c

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

fix: adapt tests of the schemas

10699ea

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

feat: add option to open default editor

ed5f601

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

Merge branch 'feature/reporting/save-reporters-data-source' into feat…

68c2c00

…ure/reporting/add-report-support-yml

victorgarcia98 added Still Needs Changelog Entry Reporting labels Jun 28, 2023

victorgarcia98 changed the title ~~feat: support YAML in flexmeasures add report comman~~ feat: support YAML in flexmeasures add report command Jun 28, 2023

victorgarcia98 self-assigned this Jun 28, 2023

Merge branch 'feature/reporting/save-reporters-data-source' into feat…

552a858

…ure/reporting/add-report-support-yml Signed-off-by: Victor <victor@seita.nl>

victorgarcia98 marked this pull request as ready for review June 28, 2023 13:03

victorgarcia98 requested a review from Flix6x June 28, 2023 13:06

Merge branch 'feature/reporting/save-reporters-data-source' into feat…

5f152fb

…ure/reporting/add-report-support-yml Signed-off-by: Victor <victor@seita.nl>

Flix6x requested changes Jul 2, 2023

View reviewed changes

victorgarcia98 added 3 commits July 3, 2023 14:19

Merge branch 'feature/reporting/save-reporters-data-source' into feat…

baa4a4e

…ure/reporting/add-report-support-yml Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

fix: move sensor to input

e4fb7df

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

Merge remote-tracking branch 'origin/feature/reporting/add-report-sup…

150be22

…port-yml' into feature/reporting/add-report-support-yml Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

victorgarcia98 requested a review from Flix6x July 3, 2023 17:55

nhoening added this to the 0.15.0 milestone Jul 4, 2023

Flix6x requested changes Jul 6, 2023

View reviewed changes

victorgarcia98 added 3 commits July 7, 2023 16:25

fix: parse resolution properly

1d6f231

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

fix: remove accidentally commited file

7a67ba4

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

fix: avoid potential bug

1961ebf

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

victorgarcia98 added 2 commits August 2, 2023 21:16

Merge branch 'feature/reporting/save-reporters-data-source' into feat…

a4b7965

…ure/reporting/add-report-support-yml Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

rename input to parameters

edb67af

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

add chagelog entry

9b27504

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

victorgarcia98 requested a review from Flix6x August 2, 2023 19:59

add pyyaml to app.txt

f8990bb

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

victorgarcia98 removed the Still Needs Changelog Entry label Aug 3, 2023

victorgarcia98 added 3 commits August 3, 2023 11:58

Merge branch 'feature/reporting/save-reporters-data-source' into feat…

9ea66ec

…ure/reporting/add-report-support-yml Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

Merge branch 'feature/reporting/save-reporters-data-source' into feat…

f19494f

…ure/reporting/add-report-support-yml

add --save-config to the add_report command

8908bfc

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

Flix6x approved these changes Aug 3, 2023

View reviewed changes

documentation/changelog.rst Outdated Show resolved Hide resolved

documentation/cli/change_log.rst Outdated Show resolved Hide resolved

victorgarcia98 added 2 commits August 3, 2023 23:38

improve changelog files

218d6a2

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

Merge branch 'feature/reporting/save-reporters-data-source' into feat…

1c1c37e

…ure/reporting/add-report-support-yml Signed-off-by: Victor <victor@seita.nl>

victorgarcia98 merged commit 1e8704e into feature/reporting/save-reporters-data-source Aug 3, 2023
4 checks passed

victorgarcia98 deleted the feature/reporting/add-report-support-yml branch August 3, 2023 21:47

Flix6x mentioned this pull request Aug 3, 2023

feat: save/fetch Reporter as DataSource #751

Merged

victorgarcia98 mentioned this pull request Aug 3, 2023

Polish PandasReporter schemas #788

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: support YAML in `flexmeasures add report` command #752

feat: support YAML in `flexmeasures add report` command #752

victorgarcia98 commented Jun 28, 2023 •

edited by Flix6x

Flix6x Jul 2, 2023

victorgarcia98 Jul 3, 2023

Flix6x Jul 6, 2023

Flix6x left a comment

Flix6x Jul 6, 2023

Flix6x commented Aug 1, 2023

Flix6x commented Aug 1, 2023

victorgarcia98 commented Aug 2, 2023

Flix6x left a comment

feat: support YAML in flexmeasures add report command #752

feat: support YAML in flexmeasures add report command #752

Conversation

victorgarcia98 commented Jun 28, 2023 • edited by Flix6x

Flix6x Jul 2, 2023

Choose a reason for hiding this comment

victorgarcia98 Jul 3, 2023

Choose a reason for hiding this comment

Flix6x Jul 6, 2023

Choose a reason for hiding this comment

Flix6x left a comment

Choose a reason for hiding this comment

Flix6x Jul 6, 2023

Choose a reason for hiding this comment

Flix6x commented Aug 1, 2023

Flix6x commented Aug 1, 2023

victorgarcia98 commented Aug 2, 2023

Flix6x left a comment

Choose a reason for hiding this comment

feat: support YAML in `flexmeasures add report` command #752

feat: support YAML in `flexmeasures add report` command #752

victorgarcia98 commented Jun 28, 2023 •

edited by Flix6x