fix: drop NaN values when saving the report to the database #735
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
Signed-off-by: F.N. Claessen <felix@seita.nl>
victorgarcia98
approved these changes
Jun 20, 2023
There was a problem hiding this comment.
Thanks!
Not sure about moving the dropna to save_to_db.
For instance, in some places could have another column for the values with some NaN and dropping the NaN would drop, by default, any column having any NaN.
To fix this, we should use subset=["event_value"] to make sure that we only take the column event_value into account.
Regarding the the use case of storing NaN to the database, I think that as long as we can save a database with an irregular sampling rate, we are good to go.
Flix6x
added a commit
that referenced
this pull request
Jun 26, 2023
…database (#735) * fix: drop NaN values when saving the report to the database Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: changelog entry Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: explain abbreviation Signed-off-by: F.N. Claessen <felix@seita.nl> * docs: CLI changelog entry Signed-off-by: F.N. Claessen <felix@seita.nl> --------- Signed-off-by: F.N. Claessen <felix@seita.nl>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I'm actually considering moving this logic (of dropping NaNs before saving the data to the database) into
save_to_db, but I feel that might constitute a more impactful change then needed for this fix. What do you think, @victorgarcia98? Do you see any use for supporting saving NaN values to the database?For anyone impacted by this: there already exists a CLI command for removing NaN values from the database: see
flexmeasures delete nan-beliefs.