Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Polish PandasReporter schemas #788

Merged
merged 13 commits into from Aug 7, 2023
Merged

Polish PandasReporter schemas #788

merged 13 commits into from Aug 7, 2023

Conversation

victorgarcia98
Copy link
Contributor

@victorgarcia98 victorgarcia98 commented Aug 3, 2023 •
edited by Flix6x

Description

This PR polishes the schemas of the PandasReporter.

In the PandasReporter schemas:

  • Replace input_variables with required_input
  • Replace final_df_output with required_ouput
  • Replace input_sensors with input
  • Add output field to define multiple outputs

In the AggregatorReporter:

  • Replace data by input

How to test

pytest -k 'not test_add_report'

Further Improvements

Support returning multiple outputs in the compute and save them to the DB.

My suggestion is to return a dictionary that uses the output but add the results BDF of each output "channel".

Related Items

PRs: #751 #752

  • I agree to contribute to the project under Apache 2 License.
  • To the best of my knowledge, the proposed patch is not based on code under GPL or other license that is incompatible with FlexMeasures

@victorgarcia98
add required_input and required_output to PandasReporterConfigSchema …
0f80fce 
…and input & outupt to parameters

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>
@victorgarcia98
Merge remote-tracking branch 'origin/feature/reporting/save-reporters…
9c1c1e5 
…-data-source' into feature/reporting/adapt-pandas-reporter-to-standard-io-schema
@victorgarcia98
adapt tests
8e2ac06 
Signed-off-by: Victor Garcia Reolid <victor@seita.nl>
@victorgarcia98 victorgarcia98 self-assigned this Aug 3, 2023
@victorgarcia98 victorgarcia98 mentioned this pull request Aug 3, 2023
Merged
2 tasks
Flix6x
Flix6x requested changes Aug 4, 2023
View reviewed changes
Copy link
Contributor

@Flix6x Flix6x left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very nice. I feel this takes it from great to excellent! Let's expand the changelog entry for #751 to include this PR number.

Minor comments only, but I'd like to give it one last check after you addressed them.

flexmeasures/data/schemas/reporting/aggregation.py Show resolved Hide resolved
flexmeasures/data/models/reporting/aggregator.py Outdated Show resolved Hide resolved
flexmeasures/data/schemas/io.py Outdated Show resolved Hide resolved
flexmeasures/data/schemas/reporting/pandas_reporter.py Outdated Show resolved Hide resolved
flexmeasures/data/schemas/reporting/pandas_reporter.py Outdated Show resolved Hide resolved
flexmeasures/data/models/reporting/pandas_reporter.py Outdated Show resolved Hide resolved
flexmeasures/data/models/reporting/pandas_reporter.py Show resolved Hide resolved
flexmeasures/data/models/reporting/pandas_reporter.py Show resolved Hide resolved
flexmeasures/data/models/reporting/pandas_reporter.py Outdated Show resolved Hide resolved
flexmeasures/data/models/reporting/pandas_reporter.py Outdated Show resolved Hide resolved
@victorgarcia98
implement multiple output and simplify tibber reporter
b7a52aa 
Signed-off-by: Victor Garcia Reolid <victor@seita.nl>
Flix6x
Flix6x requested changes Aug 5, 2023
View reviewed changes
Copy link
Contributor

@Flix6x Flix6x left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do the max constraints on the output still need to be removed?

flexmeasures/data/models/data_sources.py Show resolved Hide resolved
flexmeasures/data/models/data_sources.py Outdated Show resolved Hide resolved
flexmeasures/data/schemas/reporting/aggregation.py Outdated Show resolved Hide resolved
flexmeasures/data/schemas/reporting/aggregation.py Show resolved Hide resolved
flexmeasures/data/schemas/reporting/pandas_reporter.py Outdated Show resolved Hide resolved
flexmeasures/data/schemas/reporting/pandas_reporter.py Show resolved Hide resolved
flexmeasures/data/tests/test_data_source.py Outdated Show resolved Hide resolved
flexmeasures/data/schemas/reporting/__init__.py Outdated Show resolved Hide resolved
@victorgarcia98
fix example in the docstring
ce763c6 
Signed-off-by: Victor Garcia Reolid <victor@seita.nl>
@victorgarcia98
remove max=1 constraint
44d53fb 
Signed-off-by: Victor Garcia Reolid <victor@seita.nl>
@victorgarcia98
add example for _clean_parameters
17120b6 
Signed-off-by: Victor Garcia Reolid <victor@seita.nl>
@victorgarcia98
remove time parameters in input (_clean_parameters method)
1c01027 
Signed-off-by: Victor Garcia Reolid <victor@seita.nl>
@victorgarcia98
remove filed added accidentally
be3fd42 
Signed-off-by: Victor Garcia Reolid <victor@seita.nl>
@victorgarcia98
improve assert
2d9ea2b 
Signed-off-by: Victor Garcia Reolid <victor@seita.nl>
@Flix6x
Copy link
Contributor

Flix6x commented Aug 6, 2023

Just still expand the changelog entry for #751 to include this PR number.

@victorgarcia98
update changelog entry
6d2219d 
Signed-off-by: Victor Garcia Reolid <victor@seita.nl>
victorgarcia98 and others added 2 commits August 7, 2023 09:03
@victorgarcia98
fix changelog
a6829d1 
Signed-off-by: Victor Garcia Reolid <victor@seita.nl>
@victorgarcia98 @Flix6x
Adapt test_add_report to use the new field of the PandasReporter …
5bf20ac 
…schema (#789)

* fix: typo

Signed-off-by: F.N. Claessen <felix@seita.nl>

* fix: fetch output sensor only from parameters dict

Signed-off-by: F.N. Claessen <felix@seita.nl>

* adapt the CLI to deal with multiple output

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix typos

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

---------

Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: Victor Garcia Reolid <victor@seita.nl>
Co-authored-by: F.N. Claessen <felix@seita.nl>
@victorgarcia98 victorgarcia98 merged commit 0a38d76 into feature/reporting/save-reporters-data-source Aug 7, 2023
1 of 2 checks passed
@victorgarcia98 victorgarcia98 deleted the feature/reporting/adapt-pandas-reporter-to-standard-io-schema branch August 7, 2023 07:47
victorgarcia98 added a commit that referenced this pull request Aug 7, 2023
@victorgarcia98 @Flix6x
feat: save/fetch Reporter as DataSource (#751)
2866237 
* feat: revision to add `attributes` column to the `data_source` table

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* feat: add `attributes` column to the DataSource model

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* feat: add sensors relationship in DataSource

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: make sensors relationship viewonly

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* feat: add report_config to Reporter class

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* feat: add PandasReporter report and reporter schemas

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: update fixture by removing beliefs_search_configs and adding input_variables

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* feat: add report config to PandasReporter

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* feat: add helper methods to DataSource

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: modernize AggregatorReporter

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* feat: add attributes hash

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* feat: add attributes to the function get_or_create_source

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* feat: add attribute hash to get_or_create_source

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* feat: save/fetch data generator to/from data source

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* refactor: adapt reporters to use new DataGenerator class

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: use default method on load

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: adapt tests of the schemas

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: use a DataGenerator with a schema defined

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* changing backref from "dynamic" to "select"

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* feat: add hash_attributes static method

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: use hash_attributes static method

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* feat: adding attributes_hash to the DataSource unique constraint list

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: add constraint to migration and downgrade

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: only returning keys from the attributes field

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* refactor: rename _inputs_schema to _input_schema

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: typing

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: avoid future data leakage

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* refactor: rename PandasReporterInputConfigSchema to PandasReporterInputSchema

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* docs: clarify description of the fake_data mock variable

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* docs: fix docstring

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: use default value

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: allow creating new attributes with the method `set_attributes`

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* docs: add changelog entry

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* docs: fix docstring

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: use default value

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: allow creating new attributes with the method `set_attributes`

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* docs: add changelog entry

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: add reporters and schedulers into the data_generators attribute in the app context

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: raise Exceptions instead of returning None

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: move sensor attribute from config to inputs

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: use same structure for data generators and add test

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* refactor: use input_resolution instead of resolution

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* doc: update schema docstring

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* refactor: rename input_resolution to resolution

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: remove sensor from config

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* docs: add comment

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: remove df_output

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix:. for data in data["data"] haha

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* doc: add docstring to compute and __init__ in DataGenerator

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* refactor: rename inputs to input

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: removing constructor

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* docs: improve docstring

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* test: add data to confest

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* test: add test_dst_transition

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: never returning None

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* test: add test to check timely-beliefs resampling and calling an aggregator between to dates with different offsets, in this case, daylight saving transitions.

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* test: change output sensor id

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* docs: add docstring to the data_source propert of the class DataGenerator.

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* docs: edit data_source docstring

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* refactor: ranming input_sensors to input_variables

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* rename input to parameters

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* remove unnecessary import

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* add save_config attribute

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* remove leftover comment

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* add inline comments

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* deprecation message for app.reoprters and app.schedulers

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix typo

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* use data source created by  the data generator

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* feat: support YAML in `flexmeasures add report` command (#752)

* feat: add pyyaml to the requirements

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* feat: support YAML and add report_config

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: move `types-PyYAML` dependency to the right place

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: use a DataGenerator with defined schemas

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: adapt tests of the schemas

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* feat: add option to open default editor

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: move sensor to input

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: parse resolution properly

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: remove accidentally commited file

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix: avoid potential bug

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* rename input to parameters

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* add chagelog entry

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* add pyyaml to app.txt

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* add --save-config to the add_report command

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* improve changelog files

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

---------

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>
Signed-off-by: Victor <victor@seita.nl>

* Polish `PandasReporter` schemas (#788)

* add required_input and required_output to PandasReporterConfigSchema and input & outupt to parameters

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* adapt tests

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* implement multiple output and simplify tibber reporter

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix example in the docstring

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* remove max=1 constraint

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* add example for _clean_parameters

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* remove time parameters in input (_clean_parameters method)

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* remove filed added accidentally

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* improve assert

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* update changelog entry

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix changelog

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* Adapt `test_add_report` to use the new field of the `PandasReporter` schema (#789)

* fix: typo

Signed-off-by: F.N. Claessen <felix@seita.nl>

* fix: fetch output sensor only from parameters dict

Signed-off-by: F.N. Claessen <felix@seita.nl>

* adapt the CLI to deal with multiple output

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

* fix typos

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>

---------

Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: Victor Garcia Reolid <victor@seita.nl>
Co-authored-by: F.N. Claessen <felix@seita.nl>

---------

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Co-authored-by: F.N. Claessen <felix@seita.nl>

---------

Signed-off-by: Victor Garcia Reolid <victor@seita.nl>
Signed-off-by: Victor <victor@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Co-authored-by: F.N. Claessen <felix@seita.nl>
@Flix6x Flix6x added this to the 0.15.0 milestone Aug 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Flix6x Flix6x Flix6x approved these changes

Assignees

@victorgarcia98 victorgarcia98

Labels
Reporting
Projects
None yet
Milestone
0.15.0
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@victorgarcia98 @Flix6x