New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Issue 284 Move sensor data from Power/Price/Weather to TimedBelief #286
Issue 284 Move sensor data from Power/Price/Weather to TimedBelief #286
Commits on Dec 23, 2021
-
Sub issue 284a initialize timed belief when initializing power/price/…
…weather (#285) Ensure that new data is saved in both the old (Power/Price/Weather) and the new model (TimedBelief). Resolve interference between some tests using day-ahead price data by adjusting the data source of the price data for tests using the new model. * Initialize TimedBelief when initializing Weather Signed-off-by: F.N. Claessen <felix@seita.nl> * Add deprecation notice for Weather class Signed-off-by: F.N. Claessen <felix@seita.nl> * Initialize TimedBelief when initializing Price, and add deprecation notice for Price class Signed-off-by: F.N. Claessen <felix@seita.nl> * Fix test data periodicity Signed-off-by: F.N. Claessen <felix@seita.nl> * Resolve test interference by using a different data source in conftest Signed-off-by: F.N. Claessen <felix@seita.nl> * Initialize TimedBelief when initializing Power, and add deprecation notice for Power class Signed-off-by: F.N. Claessen <felix@seita.nl> * Initialize TimedBelief when initializing forecasts Signed-off-by: F.N. Claessen <felix@seita.nl> * Update incorrect test: a shorter scheduling horizon may lead to a different result Signed-off-by: F.N. Claessen <felix@seita.nl>
Commits on Dec 29, 2021
-
Sub issue 284b query timed belief rather than power/price/weather (#287)
Move time series queries to the new data model, and also have forecasting and scheduling jobs directly create TimedBeliefs. * Query TimedBelief rather than Power in api v1.3 tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in api v1.3 implementations Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in user services tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in query tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in forecasting tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in scheduling tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in api v1 tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Simplify data deletion, like, by a lot Signed-off-by: F.N. Claessen <felix@seita.nl> * Count ex-ante TimedBeliefs after populating time series forecasts Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Price in api v1_1 tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power/Price/Weather in Resource.load_sensor_data Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power/Price/Weather in api v2.0 tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Refactor: simplify duplicate query construction Signed-off-by: F.N. Claessen <felix@seita.nl> * Add custom join target to get rid of SA warning Signed-off-by: F.N. Claessen <felix@seita.nl> * Filter criteria should work for both TimedBeliefs and TimedValues Signed-off-by: F.N. Claessen <felix@seita.nl> * Clarify docstring Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in api v1 implementations Signed-off-by: F.N. Claessen <felix@seita.nl> * Schedules should contain one deterministic belief per event Signed-off-by: F.N. Claessen <felix@seita.nl> * Fix type annotation Signed-off-by: F.N. Claessen <felix@seita.nl> * flake8 Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Price/Weather for analytics Signed-off-by: F.N. Claessen <felix@seita.nl> * Query deterministic TimedBelief rather than Price for planning queries Signed-off-by: F.N. Claessen <felix@seita.nl> * Forecast TimedBelief rather than Power/Price/Weather Signed-off-by: F.N. Claessen <felix@seita.nl> * Schedule TimedBelief rather than Power Signed-off-by: F.N. Claessen <felix@seita.nl> * Apparently, to initialize a TimedBelief is to save a TimedBelief, too Signed-off-by: F.N. Claessen <felix@seita.nl> * Create TimedBelief rather than Power/Price/Weather in data generation script Signed-off-by: F.N. Claessen <felix@seita.nl> * Bump timely-beliefs dependency Signed-off-by: F.N. Claessen <felix@seita.nl> * Fix latest state query Signed-off-by: F.N. Claessen <felix@seita.nl> * Revert "Apparently, to initialize a TimedBelief is to save a TimedBelief, too" This reverts commit fb58ec7. * Prevent saving TimedBelief to session upon updating Sensor or Source Signed-off-by: F.N. Claessen <felix@seita.nl> * Expand inline note Signed-off-by: F.N. Claessen <felix@seita.nl> * Refactor: simplify block in data_gen.py Signed-off-by: F.N. Claessen <felix@seita.nl> * Add back test and fix it by selecting only the most recent beliefs when resampling, and by solving a bug in a deprecated optional argument. Also, remove redundant index conversions in test, and move one conversion closer to where it is needed Signed-off-by: F.N. Claessen <felix@seita.nl>
-
Sub issue 284c migrate power/price/weather data to timed belief (#288)
Copy over all time series data, in batches. * Query TimedBelief rather than Power in api v1.3 tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in api v1.3 implementations Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in user services tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in query tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in forecasting tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in scheduling tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in api v1 tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Simplify data deletion, like, by a lot Signed-off-by: F.N. Claessen <felix@seita.nl> * Count ex-ante TimedBeliefs after populating time series forecasts Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Price in api v1_1 tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power/Price/Weather in Resource.load_sensor_data Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power/Price/Weather in api v2.0 tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Refactor: simplify duplicate query construction Signed-off-by: F.N. Claessen <felix@seita.nl> * Add custom join target to get rid of SA warning Signed-off-by: F.N. Claessen <felix@seita.nl> * Filter criteria should work for both TimedBeliefs and TimedValues Signed-off-by: F.N. Claessen <felix@seita.nl> * Clarify docstring Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in api v1 implementations Signed-off-by: F.N. Claessen <felix@seita.nl> * Schedules should contain one deterministic belief per event Signed-off-by: F.N. Claessen <felix@seita.nl> * Fix type annotation Signed-off-by: F.N. Claessen <felix@seita.nl> * flake8 Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Price/Weather for analytics Signed-off-by: F.N. Claessen <felix@seita.nl> * Query deterministic TimedBelief rather than Price for planning queries Signed-off-by: F.N. Claessen <felix@seita.nl> * Forecast TimedBelief rather than Power/Price/Weather Signed-off-by: F.N. Claessen <felix@seita.nl> * Schedule TimedBelief rather than Power Signed-off-by: F.N. Claessen <felix@seita.nl> * Apparently, to initialize a TimedBelief is to save a TimedBelief, too Signed-off-by: F.N. Claessen <felix@seita.nl> * Create TimedBelief rather than Power/Price/Weather in data generation script Signed-off-by: F.N. Claessen <felix@seita.nl> * Bump timely-beliefs dependency Signed-off-by: F.N. Claessen <felix@seita.nl> * Start database migration file and copy time series data Signed-off-by: F.N. Claessen <felix@seita.nl> * Remove raise statement used for debugging Signed-off-by: F.N. Claessen <felix@seita.nl> * Add changelog entry Signed-off-by: F.N. Claessen <felix@seita.nl>
Commits on Jan 3, 2022
-
Sub issue 284d stop saving to power/price/weather tables (#289)
Switch all occurrences of initializing a Power, Price or Weather object within FlexMeasures to initializing a TimedBelief instead. Also refactor the save_to_db method, moving it from the api package to the data package. Because some FlexMeasures plugins are known to depend on direct initialization of Power, Price and Weather objects, and some plugins are known to use the save_to_db method from the api package, the old functionality is left intact, with any new data automatically copied to the new model, with the appropriate deprecation warnings. * Query TimedBelief rather than Power in api v1.3 tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in api v1.3 implementations Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in user services tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in query tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in forecasting tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in scheduling tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in api v1 tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Simplify data deletion, like, by a lot Signed-off-by: F.N. Claessen <felix@seita.nl> * Count ex-ante TimedBeliefs after populating time series forecasts Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Price in api v1_1 tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power/Price/Weather in Resource.load_sensor_data Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power/Price/Weather in api v2.0 tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Refactor: simplify duplicate query construction Signed-off-by: F.N. Claessen <felix@seita.nl> * Add custom join target to get rid of SA warning Signed-off-by: F.N. Claessen <felix@seita.nl> * Filter criteria should work for both TimedBeliefs and TimedValues Signed-off-by: F.N. Claessen <felix@seita.nl> * Clarify docstring Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in api v1 implementations Signed-off-by: F.N. Claessen <felix@seita.nl> * Schedules should contain one deterministic belief per event Signed-off-by: F.N. Claessen <felix@seita.nl> * Fix type annotation Signed-off-by: F.N. Claessen <felix@seita.nl> * flake8 Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Price/Weather for analytics Signed-off-by: F.N. Claessen <felix@seita.nl> * Query deterministic TimedBelief rather than Price for planning queries Signed-off-by: F.N. Claessen <felix@seita.nl> * Forecast TimedBelief rather than Power/Price/Weather Signed-off-by: F.N. Claessen <felix@seita.nl> * Schedule TimedBelief rather than Power Signed-off-by: F.N. Claessen <felix@seita.nl> * Apparently, to initialize a TimedBelief is to save a TimedBelief, too Signed-off-by: F.N. Claessen <felix@seita.nl> * Create TimedBelief rather than Power/Price/Weather in data generation script Signed-off-by: F.N. Claessen <felix@seita.nl> * Bump timely-beliefs dependency Signed-off-by: F.N. Claessen <felix@seita.nl> * Fix latest state query Signed-off-by: F.N. Claessen <felix@seita.nl> * Revert "Apparently, to initialize a TimedBelief is to save a TimedBelief, too" This reverts commit fb58ec7. * Prevent saving TimedBelief to session upon updating Sensor or Source Signed-off-by: F.N. Claessen <felix@seita.nl> * Create only TimedBeliefs in conftests Signed-off-by: F.N. Claessen <felix@seita.nl> * Use session.add_all calls instead of session.bulk_save_objects or individual session.add calls Signed-off-by: F.N. Claessen <felix@seita.nl> * API directly creates TimedBeliefs Signed-off-by: F.N. Claessen <felix@seita.nl> * CLI uses TimedBeliefs only Signed-off-by: F.N. Claessen <felix@seita.nl> * Data scripts use TimedBeliefs only Signed-off-by: F.N. Claessen <felix@seita.nl> * One more conftest switched to creating TimedBeliefs instead of Weather objects Signed-off-by: F.N. Claessen <felix@seita.nl> * Expand docstring note on forbidden replacements Signed-off-by: F.N. Claessen <felix@seita.nl> * Clarify docstring note on saving changed beliefs only Signed-off-by: F.N. Claessen <felix@seita.nl> * Remove redundant flush Signed-off-by: F.N. Claessen <felix@seita.nl> * Catch forbidden belief replacements with more specific exception Signed-off-by: F.N. Claessen <felix@seita.nl> * Rename variable Signed-off-by: F.N. Claessen <felix@seita.nl> * One transaction per request Signed-off-by: F.N. Claessen <felix@seita.nl> * Only enqueue forecasting jobs upon successfully saving new data Signed-off-by: F.N. Claessen <felix@seita.nl> * Flush instead of commit Signed-off-by: F.N. Claessen <felix@seita.nl> * Expand test for forbidden data replacement Signed-off-by: F.N. Claessen <felix@seita.nl> * Simplify play mode excemption for replacing beliefs Signed-off-by: F.N. Claessen <felix@seita.nl> * Add note about potential session rollback Signed-off-by: F.N. Claessen <felix@seita.nl> * Typo Signed-off-by: F.N. Claessen <felix@seita.nl> * Move UniqueViolation catching logic to error handler Signed-off-by: F.N. Claessen <felix@seita.nl> * flake8 Signed-off-by: F.N. Claessen <felix@seita.nl> * Clean up Signed-off-by: F.N. Claessen <felix@seita.nl> * Refactor: move error handler to api_utils.py Signed-off-by: F.N. Claessen <felix@seita.nl> * flake8 Signed-off-by: F.N. Claessen <felix@seita.nl>
-
Merge branch 'project-9' into Issue-284_Move_sensor_data_from_Power/P…
…rice/Weather_to_TimedBelief # Conflicts: # flexmeasures/api/dev/tests/test_sensor_data.py