Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

UI for data upload #563

Draft
wants to merge 19 commits into
base: main
Choose a base branch
from
Draft

UI for data upload #563

wants to merge 19 commits into from

Conversation

Flix6x
Copy link
Contributor

@Flix6x Flix6x commented Dec 26, 2022 •
edited

A proof of concept, with a little styling. Still needs:

Follow-up issues:

  • Set data source type, e.g. forecaster or scheduler (users can be linked to multiple data sources, to discuss)
  • Select data source (possible admin feature, to attribute the data to a specific data source)

@Flix6x
Tech spike: upload CSV files and parse as pandas DataFrames
75bdd98 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Parse uploaded CSV files as BeliefsDataFrames
43b7993 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Validate sensor id
a004e17 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
UI styling for upload widget: moved to new side panel
77ed733 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Reload data after uploading
e485998 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Actually save the data
67eee73 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x Flix6x added this to the 0.13.0 milestone Dec 26, 2022
@Flix6x Flix6x self-assigned this Dec 26, 2022
@Flix6x Flix6x linked an issue Dec 26, 2022 that may be closed by this pull request
UI for data upload #7
Open
@coveralls
Copy link
Collaborator

coveralls commented Dec 26, 2022 •
edited

Pull Request Test Coverage Report for Build 3830253052

  • 37 of 72 (51.39%) changed or added relevant lines in 4 files are covered.
  • 2 unchanged lines in 1 file lost coverage.
  • Overall coverage decreased (-0.2%) to 65.358%
Changes Missing Coverage Covered Lines Changed/Added Lines %
flexmeasures/api/v3_0/sensors.py 8 10 80.0%
flexmeasures/data/schemas/utils.py 10 13 76.92%
flexmeasures/data/schemas/sensors.py 14 44 31.82%
Files with Coverage Reduction New Missed Lines %
flexmeasures/init.py 2 83.33%
Totals Coverage Status
Change from base Build 3830075233: -0.2%
Covered Lines: 6898
Relevant Lines: 9914
💛 - Coveralls

@Flix6x Flix6x mentioned this pull request Dec 27, 2022
Merged
@Flix6x
Upgrade timely-beliefs
e3f43cb 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Remove upload form from asset page (leftover from tech spike)
2f14c1f 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Deal with "empty" file list actually containing a FileStorage with an…
8610373 
… empty filename

Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Marshmallow file validation
98caefd 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Marshmallow file deserialization
1c98cb4 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Move processing of deserialized data from validation to post_load
632fbff 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Use MultiDict, because files can have duplicate keys, and we don't wa…
fa491ae 
…nt them to overwrite each other when calling .update() on the dictionary

Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Add docstrings
3abd802 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Repurpose validator to focus on the uploaded_files field only
3e02afb 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Show errors per uploaded file
a37df6e 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Processing file contents can also lead to a ValidationError
2260299 
Signed-off-by: F.N. Claessen <felix@seita.nl>Processing file contents can also lead to a ValidationError

Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x Flix6x mentioned this pull request Dec 29, 2022
Merged
7 tasks
@Flix6x
Use f-string
91ddb93 
Signed-off-by: F.N. Claessen <felix@seita.nl>
@Flix6x
Merge remote-tracking branch 'origin/main' into 7-ui-for-data-upload
2cea3c2 
Signed-off-by: F.N. Claessen <felix@seita.nl>

# Conflicts:
#	flexmeasures/api/v3_0/sensors.py
@nhoening nhoening modified the milestones: 0.13.0, 0.14.0 May 1, 2023
@Flix6x Flix6x removed this from the 0.14.0 milestone Jun 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@Flix6x Flix6x

Labels
API Still Needs Changelog Entry UI
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

UI for data upload
3 participants
@Flix6x @coveralls @nhoening