Issue 284 Move sensor data from Power/Price/Weather to TimedBelief #286
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…weather (#285) Ensure that new data is saved in both the old (Power/Price/Weather) and the new model (TimedBelief). Resolve interference between some tests using day-ahead price data by adjusting the data source of the price data for tests using the new model. * Initialize TimedBelief when initializing Weather Signed-off-by: F.N. Claessen <felix@seita.nl> * Add deprecation notice for Weather class Signed-off-by: F.N. Claessen <felix@seita.nl> * Initialize TimedBelief when initializing Price, and add deprecation notice for Price class Signed-off-by: F.N. Claessen <felix@seita.nl> * Fix test data periodicity Signed-off-by: F.N. Claessen <felix@seita.nl> * Resolve test interference by using a different data source in conftest Signed-off-by: F.N. Claessen <felix@seita.nl> * Initialize TimedBelief when initializing Power, and add deprecation notice for Power class Signed-off-by: F.N. Claessen <felix@seita.nl> * Initialize TimedBelief when initializing forecasts Signed-off-by: F.N. Claessen <felix@seita.nl> * Update incorrect test: a shorter scheduling horizon may lead to a different result Signed-off-by: F.N. Claessen <felix@seita.nl>
4 tasks
1 task
Move time series queries to the new data model, and also have forecasting and scheduling jobs directly create TimedBeliefs. * Query TimedBelief rather than Power in api v1.3 tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in api v1.3 implementations Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in user services tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in query tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in forecasting tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in scheduling tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in api v1 tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Simplify data deletion, like, by a lot Signed-off-by: F.N. Claessen <felix@seita.nl> * Count ex-ante TimedBeliefs after populating time series forecasts Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Price in api v1_1 tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power/Price/Weather in Resource.load_sensor_data Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power/Price/Weather in api v2.0 tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Refactor: simplify duplicate query construction Signed-off-by: F.N. Claessen <felix@seita.nl> * Add custom join target to get rid of SA warning Signed-off-by: F.N. Claessen <felix@seita.nl> * Filter criteria should work for both TimedBeliefs and TimedValues Signed-off-by: F.N. Claessen <felix@seita.nl> * Clarify docstring Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in api v1 implementations Signed-off-by: F.N. Claessen <felix@seita.nl> * Schedules should contain one deterministic belief per event Signed-off-by: F.N. Claessen <felix@seita.nl> * Fix type annotation Signed-off-by: F.N. Claessen <felix@seita.nl> * flake8 Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Price/Weather for analytics Signed-off-by: F.N. Claessen <felix@seita.nl> * Query deterministic TimedBelief rather than Price for planning queries Signed-off-by: F.N. Claessen <felix@seita.nl> * Forecast TimedBelief rather than Power/Price/Weather Signed-off-by: F.N. Claessen <felix@seita.nl> * Schedule TimedBelief rather than Power Signed-off-by: F.N. Claessen <felix@seita.nl> * Apparently, to initialize a TimedBelief is to save a TimedBelief, too Signed-off-by: F.N. Claessen <felix@seita.nl> * Create TimedBelief rather than Power/Price/Weather in data generation script Signed-off-by: F.N. Claessen <felix@seita.nl> * Bump timely-beliefs dependency Signed-off-by: F.N. Claessen <felix@seita.nl> * Fix latest state query Signed-off-by: F.N. Claessen <felix@seita.nl> * Revert "Apparently, to initialize a TimedBelief is to save a TimedBelief, too" This reverts commit fb58ec7. * Prevent saving TimedBelief to session upon updating Sensor or Source Signed-off-by: F.N. Claessen <felix@seita.nl> * Expand inline note Signed-off-by: F.N. Claessen <felix@seita.nl> * Refactor: simplify block in data_gen.py Signed-off-by: F.N. Claessen <felix@seita.nl> * Add back test and fix it by selecting only the most recent beliefs when resampling, and by solving a bug in a deprecated optional argument. Also, remove redundant index conversions in test, and move one conversion closer to where it is needed Signed-off-by: F.N. Claessen <felix@seita.nl>
Copy over all time series data, in batches. * Query TimedBelief rather than Power in api v1.3 tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in api v1.3 implementations Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in user services tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in query tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in forecasting tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in scheduling tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in api v1 tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Simplify data deletion, like, by a lot Signed-off-by: F.N. Claessen <felix@seita.nl> * Count ex-ante TimedBeliefs after populating time series forecasts Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Price in api v1_1 tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power/Price/Weather in Resource.load_sensor_data Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power/Price/Weather in api v2.0 tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Refactor: simplify duplicate query construction Signed-off-by: F.N. Claessen <felix@seita.nl> * Add custom join target to get rid of SA warning Signed-off-by: F.N. Claessen <felix@seita.nl> * Filter criteria should work for both TimedBeliefs and TimedValues Signed-off-by: F.N. Claessen <felix@seita.nl> * Clarify docstring Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in api v1 implementations Signed-off-by: F.N. Claessen <felix@seita.nl> * Schedules should contain one deterministic belief per event Signed-off-by: F.N. Claessen <felix@seita.nl> * Fix type annotation Signed-off-by: F.N. Claessen <felix@seita.nl> * flake8 Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Price/Weather for analytics Signed-off-by: F.N. Claessen <felix@seita.nl> * Query deterministic TimedBelief rather than Price for planning queries Signed-off-by: F.N. Claessen <felix@seita.nl> * Forecast TimedBelief rather than Power/Price/Weather Signed-off-by: F.N. Claessen <felix@seita.nl> * Schedule TimedBelief rather than Power Signed-off-by: F.N. Claessen <felix@seita.nl> * Apparently, to initialize a TimedBelief is to save a TimedBelief, too Signed-off-by: F.N. Claessen <felix@seita.nl> * Create TimedBelief rather than Power/Price/Weather in data generation script Signed-off-by: F.N. Claessen <felix@seita.nl> * Bump timely-beliefs dependency Signed-off-by: F.N. Claessen <felix@seita.nl> * Start database migration file and copy time series data Signed-off-by: F.N. Claessen <felix@seita.nl> * Remove raise statement used for debugging Signed-off-by: F.N. Claessen <felix@seita.nl> * Add changelog entry Signed-off-by: F.N. Claessen <felix@seita.nl>
Switch all occurrences of initializing a Power, Price or Weather object within FlexMeasures to initializing a TimedBelief instead. Also refactor the save_to_db method, moving it from the api package to the data package. Because some FlexMeasures plugins are known to depend on direct initialization of Power, Price and Weather objects, and some plugins are known to use the save_to_db method from the api package, the old functionality is left intact, with any new data automatically copied to the new model, with the appropriate deprecation warnings. * Query TimedBelief rather than Power in api v1.3 tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in api v1.3 implementations Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in user services tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in query tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in forecasting tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in scheduling tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in api v1 tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Simplify data deletion, like, by a lot Signed-off-by: F.N. Claessen <felix@seita.nl> * Count ex-ante TimedBeliefs after populating time series forecasts Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Price in api v1_1 tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power/Price/Weather in Resource.load_sensor_data Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power/Price/Weather in api v2.0 tests Signed-off-by: F.N. Claessen <felix@seita.nl> * Refactor: simplify duplicate query construction Signed-off-by: F.N. Claessen <felix@seita.nl> * Add custom join target to get rid of SA warning Signed-off-by: F.N. Claessen <felix@seita.nl> * Filter criteria should work for both TimedBeliefs and TimedValues Signed-off-by: F.N. Claessen <felix@seita.nl> * Clarify docstring Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Power in api v1 implementations Signed-off-by: F.N. Claessen <felix@seita.nl> * Schedules should contain one deterministic belief per event Signed-off-by: F.N. Claessen <felix@seita.nl> * Fix type annotation Signed-off-by: F.N. Claessen <felix@seita.nl> * flake8 Signed-off-by: F.N. Claessen <felix@seita.nl> * Query TimedBelief rather than Price/Weather for analytics Signed-off-by: F.N. Claessen <felix@seita.nl> * Query deterministic TimedBelief rather than Price for planning queries Signed-off-by: F.N. Claessen <felix@seita.nl> * Forecast TimedBelief rather than Power/Price/Weather Signed-off-by: F.N. Claessen <felix@seita.nl> * Schedule TimedBelief rather than Power Signed-off-by: F.N. Claessen <felix@seita.nl> * Apparently, to initialize a TimedBelief is to save a TimedBelief, too Signed-off-by: F.N. Claessen <felix@seita.nl> * Create TimedBelief rather than Power/Price/Weather in data generation script Signed-off-by: F.N. Claessen <felix@seita.nl> * Bump timely-beliefs dependency Signed-off-by: F.N. Claessen <felix@seita.nl> * Fix latest state query Signed-off-by: F.N. Claessen <felix@seita.nl> * Revert "Apparently, to initialize a TimedBelief is to save a TimedBelief, too" This reverts commit fb58ec7. * Prevent saving TimedBelief to session upon updating Sensor or Source Signed-off-by: F.N. Claessen <felix@seita.nl> * Create only TimedBeliefs in conftests Signed-off-by: F.N. Claessen <felix@seita.nl> * Use session.add_all calls instead of session.bulk_save_objects or individual session.add calls Signed-off-by: F.N. Claessen <felix@seita.nl> * API directly creates TimedBeliefs Signed-off-by: F.N. Claessen <felix@seita.nl> * CLI uses TimedBeliefs only Signed-off-by: F.N. Claessen <felix@seita.nl> * Data scripts use TimedBeliefs only Signed-off-by: F.N. Claessen <felix@seita.nl> * One more conftest switched to creating TimedBeliefs instead of Weather objects Signed-off-by: F.N. Claessen <felix@seita.nl> * Expand docstring note on forbidden replacements Signed-off-by: F.N. Claessen <felix@seita.nl> * Clarify docstring note on saving changed beliefs only Signed-off-by: F.N. Claessen <felix@seita.nl> * Remove redundant flush Signed-off-by: F.N. Claessen <felix@seita.nl> * Catch forbidden belief replacements with more specific exception Signed-off-by: F.N. Claessen <felix@seita.nl> * Rename variable Signed-off-by: F.N. Claessen <felix@seita.nl> * One transaction per request Signed-off-by: F.N. Claessen <felix@seita.nl> * Only enqueue forecasting jobs upon successfully saving new data Signed-off-by: F.N. Claessen <felix@seita.nl> * Flush instead of commit Signed-off-by: F.N. Claessen <felix@seita.nl> * Expand test for forbidden data replacement Signed-off-by: F.N. Claessen <felix@seita.nl> * Simplify play mode excemption for replacing beliefs Signed-off-by: F.N. Claessen <felix@seita.nl> * Add note about potential session rollback Signed-off-by: F.N. Claessen <felix@seita.nl> * Typo Signed-off-by: F.N. Claessen <felix@seita.nl> * Move UniqueViolation catching logic to error handler Signed-off-by: F.N. Claessen <felix@seita.nl> * flake8 Signed-off-by: F.N. Claessen <felix@seita.nl> * Clean up Signed-off-by: F.N. Claessen <felix@seita.nl> * Refactor: move error handler to api_utils.py Signed-off-by: F.N. Claessen <felix@seita.nl> * flake8 Signed-off-by: F.N. Claessen <felix@seita.nl>
…rice/Weather_to_TimedBelief # Conflicts: # flexmeasures/api/dev/tests/test_sensor_data.py
Flix6x
deleted the
Issue-284_Move_sensor_data_from_Power/Price/Weather_to_TimedBelief
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Draft PR with the intention to (not squash) commit to the project-9 branch.