Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Read csv with naive data, column filters and datetime/timedelta units #521

Merged
merged 26 commits into from Nov 10, 2022

Conversation

Flix6x
Copy link
Contributor

@Flix6x Flix6x commented Oct 31, 2022

Closes #507, and introduces additional CSV import features.

Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
…ut a timezone set explicitly

Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x Flix6x added the enhancement New feature or request label Oct 31, 2022
@Flix6x Flix6x self-assigned this Oct 31, 2022
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x Flix6x added the CLI label Oct 31, 2022
@Flix6x Flix6x marked this pull request as ready for review October 31, 2022 18:33
…source type

Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
@coveralls
Copy link
Collaborator

coveralls commented Oct 31, 2022

Pull Request Test Coverage Report for Build 3439649207

  • 9 of 29 (31.03%) changed or added relevant lines in 3 files are covered.
  • 1 unchanged line in 1 file lost coverage.
  • Overall coverage decreased (-0.06%) to 64.967%

Changes Missing Coverage Covered Lines Changed/Added Lines %
flexmeasures/data/queries/data_sources.py 1 4 25.0%
flexmeasures/utils/unit_utils.py 3 9 33.33%
flexmeasures/cli/data_add.py 5 16 31.25%
Files with Coverage Reduction New Missed Lines %
flexmeasures/cli/data_add.py 1 31.39%
Totals Coverage Status
Change from base Build 3436864090: -0.06%
Covered Lines: 6570
Relevant Lines: 9491

💛 - Coveralls

@Flix6x Flix6x requested a review from nhoening October 31, 2022 19:31
Copy link
Contributor

@nhoening nhoening left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks useful. As so often, I want some documentation improvements.

flexmeasures/cli/data_add.py Outdated Show resolved Hide resolved
documentation/cli/change_log.rst Outdated Show resolved Hide resolved
flexmeasures/utils/unit_utils.py Outdated Show resolved Hide resolved
Signed-off-by: F.N. Claessen <felix@seita.nl>
…ata-and-filters

# Conflicts:
#	documentation/changelog.rst
#	documentation/cli/change_log.rst
Signed-off-by: F.N. Claessen <felix@seita.nl>
…since UNIX epoch)

Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
… digit

Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x Flix6x requested a review from nhoening November 10, 2022 13:45
Copy link
Contributor

@nhoening nhoening left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I approve the recent clarifications, but I feel something has gone wrong when merging?

documentation/changelog.rst Outdated Show resolved Hide resolved
documentation/changelog.rst Outdated Show resolved Hide resolved
flexmeasures/data/schemas/assets.py Outdated Show resolved Hide resolved
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Copy link
Contributor Author

Flix6x commented Nov 10, 2022

Definitely something went range with merging. I believe I now have corrected all mistakes introduced.

@Flix6x Flix6x requested a review from nhoening November 10, 2022 18:30
Copy link
Contributor

@nhoening nhoening left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, one possibly missing word and an honest question.

documentation/cli/change_log.rst Outdated Show resolved Hide resolved
documentation/cli/change_log.rst Outdated Show resolved Hide resolved
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x Flix6x merged commit b1bd21d into main Nov 10, 2022
@Flix6x Flix6x deleted the read-csv-with-naive-data-and-filters branch November 10, 2022 19:12
@Flix6x Flix6x added this to the 0.12.0 milestone Nov 10, 2022
Flix6x added a commit that referenced this pull request Nov 15, 2022
…nger assume utc, but raise instead (see PR #521)

Signed-off-by: F.N. Claessen <felix@seita.nl>
Flix6x added a commit that referenced this pull request Nov 15, 2022
…nger assume utc, but raise instead (see PR #521)

Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x Flix6x mentioned this pull request Nov 15, 2022
Flix6x added a commit that referenced this pull request Nov 15, 2022
…nger assume utc, but raise instead (since PR #521).

Fix tutorial timezone (#535)

* fix charging sensor name in tutorial

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Fix for changed default when reading in timezone naive data: we no longer assume utc, but raise instead (see PR #521)

Signed-off-by: F.N. Claessen <felix@seita.nl>

Signed-off-by: F.N. Claessen <felix@seita.nl>
Flix6x added a commit that referenced this pull request Apr 12, 2023
…context of a building with PV (#534)

Expand toy tutorial and chart functionality (incl. a more sensible use of the color encoding and data source attribute in charts), by:

- Expanding the toy tutorial with creation of solar sensor and solar forecasts from CSV
- Expanding the toy tutorial with a new schedule that takes into account solar
- Allowing nesting of sensors in the `sensors_to_show` attribute, to denote that multiple sensors should be shown together, and dealing with all of its implications (axis labels, use of colors, etc.)
- Visually distinguishing forecasts, schedules and measurements by source type rather than by belief horizon (incl. a simple db migration)
- Adding a CLI command to create a new DataSource (so that forecasts are visually presented as originating from a forecaster)



* fix charging sensor name in tutorial

Signed-off-by: F.N. Claessen <felix@seita.nl>

* update price sensor name in tutorial (prefer non-capitalized sensor names)

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Print out account ID if toy account already exists

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Fix for changed default when reading in timezone naive data: we no longer assume utc, but raise instead (see PR #521)

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add solar power sensor to toy account, and allow existing toy accounts to be updated by simply running `flexmeasures add toy-account` again

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Support JSON and Callable arguments

Signed-off-by: F.N. Claessen <felix@seita.nl>

* get or create price sensor

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Refactor to keep original order of sensor ids

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add CLI option to pass inflexible device sensors

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Expand tutorial with scheduling against solar, which limits the available headroom for the battery

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add note about resampling

Signed-off-by: F.N. Claessen <felix@seita.nl>

* black

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Allow nesting sensors_to_show to support layering multiple sensors in a single row within the multi-row asset chart

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Textual changes

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Remove print statement

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add missing timezone

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Fix call to make_schedule

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Ensure storage specs, also when make_schedule is called directly (i.e. no job)

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Legend shows which asset a sensor belongs to, too, and sensors that share a name don't share a color (note that sensors belonging to the same asset aren't allowed to have the same name)

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add CLI command for adding a data source

Signed-off-by: F.N. Claessen <felix@seita.nl>

* DB migration for source types 'forecaster' and 'scheduler'

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Attribute toy solar forecasts to a new DataSource of type 'forecaster'

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Check common cases of shared sensor types

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add version identifier and missing parenthesis

Signed-off-by: F.N. Claessen <felix@seita.nl>

* black and flake8

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Fix test (no legend in PositionFieldDef)

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Remove redundant detail encodings

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Upgrade timely-beliefs for required feature from SeitaBV/timely-beliefs#122

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Merge db revisions

Signed-off-by: F.N. Claessen <felix@seita.nl>

* black

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Speed up viz by avoiding redundant client-side transformation

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Bump vega-lite and vegaembed

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Ensure full-width bar width

Signed-off-by: F.N. Claessen <felix@seita.nl>

* black

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Update CLI scheduling command by adding `for-storage`

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Improve introduction to using `flexmeasures add source`

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Introduce follow-up solar example

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Refactor: rename "scheduling script" to "scheduler" and "forecasting script" to "forecaster"

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Clarify event start domain setting

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Raise instead of warn in case toy account already exists

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Clarify loop over kwargs

Signed-off-by: F.N. Claessen <felix@seita.nl>

* black

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Merge steps

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Clarify when we choose not to show the sensor name

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add return type annotations

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add default sensors_to_show attribute to test asset

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add test cases for bad sensors_to_show attribute

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Validate sensors_to_show attribute

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Simplify code lines

Signed-off-by: F.N. Claessen <felix@seita.nl>

* black

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Refactor: move util function

Signed-off-by: F.N. Claessen <felix@seita.nl>

* flake8

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Changelog entry plus upgrade instructions

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add documentation for nested sensors_to_show attribute

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Resolve import error

Signed-off-by: F.N. Claessen <felix@seita.nl>

* redundant space

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Resolve circular import

Signed-off-by: F.N. Claessen <felix@seita.nl>

* add a missing output style

Signed-off-by: Nicolas Höning <nicolas@seita.nl>

* Fix GH Issue #604

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Correct spacing of header markings

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add sensors_to_show attribute to tutorial printout

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Correct sensor name in tutorial

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Correct column name in tutorial

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Shift sensor ids in tutorial

Signed-off-by: F.N. Claessen <felix@seita.nl>

* fix capitalization

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Update uniplot output (looks like later versions have a more intuitive choice of y-axis ticks)

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Correct data source IDs

Signed-off-by: F.N. Claessen <felix@seita.nl>

* test

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Revert "test"

This reverts commit b393cad.

* Resolve circular import coming to light through mypy

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Generalize util function

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add documentation for new CLI command

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add documentation for new CLI option

Signed-off-by: F.N. Claessen <felix@seita.nl>

---------

Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: Nicolas Höning <nicolas@seita.nl>
Co-authored-by: Nicolas Höning <nicolas@seita.nl>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLI enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Import data with local datetimes
3 participants