Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Status page by source #1022

Open
wants to merge 91 commits into
base: main
Choose a base branch
from
Open

Status page by source #1022

wants to merge 91 commits into from

Conversation

nrozanov
Copy link
Contributor

Description

  1. Split status by data source
  2. Change staleness calculation for Forecaster / Scheduler sources (staleness should be calculated as difference between now and max event start time)

Look & Feel

  1. "//status/" page should return source key as well
  2. UI status page should include source

How to test

Is tested in test_sensor_data_schema.py (test_get_status_multi_source)
Can be tested the following way - add data from several sources to some sensor. Go to its status page and check that statuses are split by this source column + scheduler/forecaster statuses are calculated differently

Further Improvements

Related Items

Follow up for #941

  • I agree to contribute to the project under Apache 2 License.
  • To the best of my knowledge, the proposed patch is not based on code under GPL or other license that is incompatible with FlexMeasures

GustaafL added 30 commits December 19, 2023 17:14
feat: add status page url
96ed8d6 
Signed-off-by: GustaafL <guus@seita.nl>
feat: test mock status api request
6a26d86 
Signed-off-by: GustaafL <guus@seita.nl>
feat: test for status with current time
d0303db 
Signed-off-by: GustaafL <guus@seita.nl>
refactor: add prints for testing
54b95fc 
Signed-off-by: GustaafL <guus@seita.nl>
fix: use correct time window for test
6fcd161 
Signed-off-by: GustaafL <guus@seita.nl>
feat: basic UI
2747ef5 
Signed-off-by: GustaafL <guus@seita.nl>
fix: remove accidental change to asset.html
fc68895 
Signed-off-by: GustaafL <guus@seita.nl>
feat: basic UI status page
ea96bd7 
Signed-off-by: GustaafL <guus@seita.nl>
feat: add authentication to status endpoint
a0ee9f1 
Signed-off-by: GustaafL <guus@seita.nl>
fix: change test order to avoid sqlalchemy error
e9904a7 
Signed-off-by: GustaafL <guus@seita.nl>
fix: use python 3.8 syntax
c837563 
Signed-off-by: GustaafL <guus@seita.nl>
fix: test amount of assets for supplier account changed
701b628 
Signed-off-by: GustaafL <guus@seita.nl>
fix: UI buttons clickable and add constants
125101c 
Signed-off-by: GustaafL <guus@seita.nl>
feat: use larger resolution in test to avoid failing chances
a34c849 
Signed-off-by: GustaafL <guus@seita.nl>
docs: add docstrings and descriptions
498e60a 
Signed-off-by: GustaafL <guus@seita.nl>
fix(UI): add id to sensors to fix link to sensor page
69ee0f6 
Signed-off-by: GustaafL <guus@seita.nl>
fix(UI): clean up UI html
83bab76 
Signed-off-by: GustaafL <guus@seita.nl>
feat(test): add test for price sensor status
4516ed1 
Signed-off-by: GustaafL <guus@seita.nl>
feat(test): refactor tests and time functions
8fc6130 
Signed-off-by: GustaafL <guus@seita.nl>
feat(status): refactor status functions
5580c44 
Signed-off-by: GustaafL <guus@seita.nl>
fix(tests): remove old tests and conftest setup
29ca558 
Signed-off-by: GustaafL <guus@seita.nl>
refactor: revert changes from earlier itteration
64216e2 
Signed-off-by: GustaafL <guus@seita.nl>
refactor: remove white line
7fab3cb 
Signed-off-by: GustaafL <guus@seita.nl>
refactor: remove unused time utils functions
39d200a 
Signed-off-by: GustaafL <guus@seita.nl>
feat(sensor): use staleness
306ca70 
Signed-off-by: GustaafL <guus@seita.nl>
feat(sensor): refactor tests and function
09033e7 
Signed-off-by: GustaafL <guus@seita.nl>
feat(sensor): passing all teests
9311df2 
Signed-off-by: GustaafL <guus@seita.nl>
feat(sensor): fix function flow
9a144d2 
Signed-off-by: GustaafL <guus@seita.nl>
feat(sensor): add tests for status
798be79 
Signed-off-by: GustaafL <guus@seita.nl>
feat(sensor): revert changes to sensor_data.py
9576242 
Signed-off-by: GustaafL <guus@seita.nl>
nrozanov and others added 7 commits March 19, 2024 20:12
@nrozanov
Merge branch 'main' into 927-status-page
9ef1ddb
@nrozanov
Post review fixes + remove forecasting status logic
56b70d5 
Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com>
@nrozanov
Merge branch '927-status-page' of https://github.com/FlexMeasures/fle…
797346c 
…xmeasures into 927-status-page
@nrozanov
Post review fix
5ea9dac 
Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com>
@nrozanov
Remove pytest-mock from requirements/app.in
e2f1722 
Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com>
@nrozanov
Api changes
00b8a58 
Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com>
@nrozanov
Split status by data source + update staleness for forecaster / sched…
2040c35 
…uler types

Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com>
@Flix6x
Copy link
Contributor

Flix6x commented Mar 26, 2024

@nrozanov if you edit this PR and set its base to 927-status-page, the diff should become a lot more to the point, making a review easier. (Once that base branch gets merged, and that branch will automatically get closed, this PR will get automatically rebased on main again.)

@nhoening
Copy link
Contributor

Maybe that was based on my bad advice, forgot that the original branch is still open.
I'm releasing now, so we can merge the other one in a bit.

@nrozanov
merge master
6a31e80 
Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com>
@nrozanov
changelog fix
ad6b2a2 
Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com>
@nrozanov
fix for python 3.8
8a94a65 
Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com>
@nhoening nhoening added this to the 0.21 milestone Mar 27, 2024
@nrozanov
note about pandas-stubs dependency + fix mypy issues
27dbbcf 
Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com>
@nrozanov
note about pandas-stubs dependency + fix mypy issues
831bce8 
Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com>
nhoening
nhoening requested changes Mar 28, 2024
View reviewed changes
Copy link
Contributor

@nhoening nhoening left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few comments, not sure I am done with the complete review, but I can't go on right now. Hope this helps you to continue.

documentation/changelog.rst Outdated Show resolved Hide resolved
flexmeasures/data/services/sensors.py Show resolved Hide resolved
flexmeasures/data/services/sensors.py Outdated Show resolved Hide resolved
Flix6x
Flix6x requested changes Mar 28, 2024
View reviewed changes
Copy link
Contributor

@Flix6x Flix6x left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's discuss a bit offline, also.

flexmeasures/data/services/sensors.py Outdated Show resolved Hide resolved
flexmeasures/data/services/sensors.py Outdated Show resolved Hide resolved
flexmeasures/ui/crud/assets.py Outdated Show resolved Hide resolved
flexmeasures/data/services/sensors.py Show resolved Hide resolved
@nrozanov
Fixes after review + look for particular source types only
d58a37e 
Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com>
nhoening
nhoening requested changes Apr 3, 2024
View reviewed changes
flexmeasures/data/services/sensors.py Outdated Show resolved Hide resolved
documentation/changelog.rst Outdated Show resolved Hide resolved
@nrozanov
fix changelog + update staleness message
63c33f5 
Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com>
@nrozanov
merge main
94d3ead 
Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com>
@Flix6x
Copy link
Contributor

Flix6x commented Apr 7, 2024

I encountered the issue again where the status page falsely reports that no data is available from a forecaster data source type (see bottom right below). There is data from this forecaster for 22 June 2023.

image

Likewise for scheduler data source types (data is available for 18 June 2023):

image

Maybe we can take a look together tomorrow.

@nrozanov
Fix staleness reason for forecaster/scheduler when there is no future…
5b66e7f 
… data

Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com>
@nrozanov
Merge main
a5d7e00 
Signed-off-by: Nikolai Rozanov <nickolay.rozanov@gmail.com>
@nrozanov nrozanov requested a review from Flix6x April 8, 2024 12:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nhoening nhoening nhoening approved these changes

@Flix6x Flix6x Awaiting requested review from Flix6x

Requested changes must be addressed to merge this pull request.

Assignees

@nrozanov nrozanov

Labels
monitoring Still Needs Changelog Entry UI
Projects
None yet
Milestone
0.21.0
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@nrozanov @Flix6x @nhoening