Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issue 284 Move sensor data from Power/Price/Weather to TimedBelief #286

Merged
merged 5 commits into from Jan 3, 2022
Merged

Issue 284 Move sensor data from Power/Price/Weather to TimedBelief #286

merged 5 commits into from Jan 3, 2022

Commits on Dec 23, 2021

  1. Sub issue 284a initialize timed belief when initializing power/price/… 

    …weather (#285)

Ensure that new data is saved in both the old (Power/Price/Weather) and the new model (TimedBelief). Resolve interference between some tests using day-ahead price data by adjusting the data source of the price data for tests using the new model.


* Initialize TimedBelief when initializing Weather

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add deprecation notice for Weather class

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Initialize TimedBelief when initializing Price, and add deprecation notice for Price class

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Fix test data periodicity

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Resolve test interference by using a different data source in conftest

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Initialize TimedBelief when initializing Power, and add deprecation notice for Power class

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Initialize TimedBelief when initializing forecasts

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Update incorrect test: a shorter scheduling horizon may lead to a different result

Signed-off-by: F.N. Claessen <felix@seita.nl>
    @Flix6x
    Flix6x committed Dec 23, 2021
    Copy the full SHA
    8f78b20 View commit details
    Browse the repository at this point in the history

Commits on Dec 29, 2021

  1. Sub issue 284b query timed belief rather than power/price/weather (#287) 

    Move time series queries to the new data model, and also have forecasting and scheduling jobs directly create TimedBeliefs.


* Query TimedBelief rather than Power in api v1.3 tests

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Power in api v1.3 implementations

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Power in user services tests

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Power in query tests

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Power in forecasting tests

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Power in scheduling tests

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Power in api v1 tests

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Simplify data deletion, like, by a lot

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Count ex-ante TimedBeliefs after populating time series forecasts

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Price in api v1_1 tests

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Power/Price/Weather in Resource.load_sensor_data

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Power/Price/Weather in api v2.0 tests

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Refactor: simplify duplicate query construction

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add custom join target to get rid of SA warning

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Filter criteria should work for both TimedBeliefs and TimedValues

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Clarify docstring

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Power in api v1 implementations

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Schedules should contain one deterministic belief per event

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Fix type annotation

Signed-off-by: F.N. Claessen <felix@seita.nl>

* flake8

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Price/Weather for analytics

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query deterministic TimedBelief rather than Price for planning queries

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Forecast TimedBelief rather than Power/Price/Weather

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Schedule TimedBelief rather than Power

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Apparently, to initialize a TimedBelief is to save a TimedBelief, too

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Create TimedBelief rather than Power/Price/Weather in data generation script

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Bump timely-beliefs dependency

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Fix latest state query

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Revert "Apparently, to initialize a TimedBelief is to save a TimedBelief, too"

This reverts commit fb58ec7.

* Prevent saving TimedBelief to session upon updating Sensor or Source

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Expand inline note

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Refactor: simplify block in data_gen.py

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add back test and fix it by selecting only the most recent beliefs when resampling, and by solving a bug in a deprecated optional argument. Also, remove redundant index conversions in test, and move one conversion closer to where it is needed

Signed-off-by: F.N. Claessen <felix@seita.nl>
    @Flix6x
    Flix6x committed Dec 29, 2021
    Copy the full SHA
    f651eea View commit details
    Browse the repository at this point in the history

  2. Sub issue 284c migrate power/price/weather data to timed belief (#288) 

    Copy over all time series data, in batches.


* Query TimedBelief rather than Power in api v1.3 tests

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Power in api v1.3 implementations

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Power in user services tests

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Power in query tests

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Power in forecasting tests

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Power in scheduling tests

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Power in api v1 tests

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Simplify data deletion, like, by a lot

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Count ex-ante TimedBeliefs after populating time series forecasts

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Price in api v1_1 tests

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Power/Price/Weather in Resource.load_sensor_data

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Power/Price/Weather in api v2.0 tests

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Refactor: simplify duplicate query construction

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add custom join target to get rid of SA warning

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Filter criteria should work for both TimedBeliefs and TimedValues

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Clarify docstring

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Power in api v1 implementations

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Schedules should contain one deterministic belief per event

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Fix type annotation

Signed-off-by: F.N. Claessen <felix@seita.nl>

* flake8

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Price/Weather for analytics

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query deterministic TimedBelief rather than Price for planning queries

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Forecast TimedBelief rather than Power/Price/Weather

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Schedule TimedBelief rather than Power

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Apparently, to initialize a TimedBelief is to save a TimedBelief, too

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Create TimedBelief rather than Power/Price/Weather in data generation script

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Bump timely-beliefs dependency

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Start database migration file and copy time series data

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Remove raise statement used for debugging

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add changelog entry

Signed-off-by: F.N. Claessen <felix@seita.nl>
    @Flix6x
    Flix6x committed Dec 29, 2021
    Copy the full SHA
    4fb564b View commit details
    Browse the repository at this point in the history

Commits on Jan 3, 2022

  1. Sub issue 284d stop saving to power/price/weather tables (#289) 

    Switch all occurrences of initializing a Power, Price or Weather object within FlexMeasures to initializing a TimedBelief instead. Also refactor the save_to_db method, moving it from the api package to the data package. Because some FlexMeasures plugins are known to depend on direct initialization of Power, Price and Weather objects, and some plugins are known to use the save_to_db method from the api package, the old functionality is left intact, with any new data automatically copied to the new model, with the appropriate deprecation warnings.


* Query TimedBelief rather than Power in api v1.3 tests

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Power in api v1.3 implementations

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Power in user services tests

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Power in query tests

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Power in forecasting tests

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Power in scheduling tests

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Power in api v1 tests

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Simplify data deletion, like, by a lot

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Count ex-ante TimedBeliefs after populating time series forecasts

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Price in api v1_1 tests

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Power/Price/Weather in Resource.load_sensor_data

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Power/Price/Weather in api v2.0 tests

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Refactor: simplify duplicate query construction

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add custom join target to get rid of SA warning

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Filter criteria should work for both TimedBeliefs and TimedValues

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Clarify docstring

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Power in api v1 implementations

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Schedules should contain one deterministic belief per event

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Fix type annotation

Signed-off-by: F.N. Claessen <felix@seita.nl>

* flake8

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query TimedBelief rather than Price/Weather for analytics

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Query deterministic TimedBelief rather than Price for planning queries

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Forecast TimedBelief rather than Power/Price/Weather

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Schedule TimedBelief rather than Power

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Apparently, to initialize a TimedBelief is to save a TimedBelief, too

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Create TimedBelief rather than Power/Price/Weather in data generation script

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Bump timely-beliefs dependency

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Fix latest state query

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Revert "Apparently, to initialize a TimedBelief is to save a TimedBelief, too"

This reverts commit fb58ec7.

* Prevent saving TimedBelief to session upon updating Sensor or Source

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Create only TimedBeliefs in conftests

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Use session.add_all calls instead of session.bulk_save_objects or individual session.add calls

Signed-off-by: F.N. Claessen <felix@seita.nl>

* API directly creates TimedBeliefs

Signed-off-by: F.N. Claessen <felix@seita.nl>

* CLI uses TimedBeliefs only

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Data scripts use TimedBeliefs only

Signed-off-by: F.N. Claessen <felix@seita.nl>

* One more conftest switched to creating TimedBeliefs instead of Weather objects

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Expand docstring note on forbidden replacements

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Clarify docstring note on saving changed beliefs only

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Remove redundant flush

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Catch forbidden belief replacements with more specific exception

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Rename variable

Signed-off-by: F.N. Claessen <felix@seita.nl>

* One transaction per request

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Only enqueue forecasting jobs upon successfully saving new data

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Flush instead of commit

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Expand test for forbidden data replacement

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Simplify play mode excemption for replacing beliefs

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Add note about potential session rollback

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Typo

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Move UniqueViolation catching logic to error handler

Signed-off-by: F.N. Claessen <felix@seita.nl>

* flake8

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Clean up

Signed-off-by: F.N. Claessen <felix@seita.nl>

* Refactor: move error handler to api_utils.py

Signed-off-by: F.N. Claessen <felix@seita.nl>

* flake8

Signed-off-by: F.N. Claessen <felix@seita.nl>
    @Flix6x
    Flix6x committed Jan 3, 2022
    Copy the full SHA
    5ba1dc7 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'project-9' into Issue-284_Move_sensor_data_from_Power/P… 

    …rice/Weather_to_TimedBelief

# Conflicts:
#	flexmeasures/api/dev/tests/test_sensor_data.py
    @Flix6x
    Flix6x committed Jan 3, 2022
    Copy the full SHA
    dd54ebc View commit details
    Browse the repository at this point in the history