feat: new chart type for sensor data: daily heatmap

[Flix6x]

This is essentially a heatmap of the sensor's values, with dates on the y-axis and time of day on the x-axis. For individual devices, it gives an insight into the device's running times. A new button lets users switch between charts.

I needed this trick to align the date labels.

[nhoening]

Maybe the name "Runtime Matrix" is better?

[Flix6x]

Maybe the name "Runtime Matrix" is better?

Suggestions welcome!

The chart type will be available for any sensor. The term "runtime" seems particularly appropriate for power sensors of individual devices. For other sensors (those recording weather, prices, or aggregate power flows like the one shown above), it seems less appropriate.

Note that, at the moment, the button text is the only user-facing term. At some point I imagine we'll also list available chart types in the CLI (flexmeasures show chart-types?) and in the chart API (documentation).

Also note that the image is already slightly outdated: in the code I've updated the button text: "Run times" is now "Time of day".

Maybe it can also help to think about chart type names in the context of imagining a longer list of chart types (for showing sensor data). On the other hand, maybe that's just over-thinking things.

[victorgarcia98]

Cool feature!!

I've been experimenting with the DST time changes to see how 23h or 25h days would look like.

• For the 23h/day case, the graph is showing an empty value.
• For the 25h/day case, the graph is overlapping values.

I'm not sure if it could be a fault on my side loading the data or there's something in the timezone location that is failing somewhere.

Code to generate the data

import asyncio

from flexmeasures_client.client import FlexMeasuresClient as Client

SENSOR_ID = 21 # 15min resolution sensor
EMAIL = "admin@admin.nl"
PASSWORD = "admin"

async def my_script():
    # Winter to Summer
    await client.post_measurements(
        sensor_id=21,
        entity_address="ea1.2023-06.localhost:fm1",
        start="2023-03-25T00:00:00+02:00",
        duration="PT80H",
        unit="MW",
        values=[1, 1, 1, 1, -1, -1, -1, -1]*40,
    )

    # Summer to Winter
    await client.post_measurements(
        sensor_id=21,
        entity_address="ea1.2023-06.localhost:fm1",
        start="2023-10-27T00:00:00+02:00",
        duration="PT80H",
        unit="MW",
        values=[1, 1, 1, 1, -1, -1, -1, -1]*40,
    )

client = Client(
    email=EMAIL,
    password=PASSWORD,
    host="localhost:5000",
)

asyncio.run(my_script())

Winter to Summer

Time Series

Matrix Chart

Summer to Winter

Time Series

Matrix Chart

[nhoening]

Okay, then "Activity Matrix" :)

Definitely follow up with API integration, as this is the way where FlexMeasures hosters can add value for their customers (who are able to use those plots in their custom frontends.
That issue therefore needs documentation, where we showcase getting plots through FlexMeasures API (I believe that is part of some other work project ...)

[Flix6x]

I've been experimenting with the DST time changes to see how 23h or 25h days would look like.

First of all, nice use of the FlexMeasures client!

Let me discuss the charts in the order you posted them. Note that the time axes are in local time.

The 23-hour day:

• The bar chart looks correct. Notice that its x-axis gets contracted around the DST transition.
• The matrix chart looks correct. There is no local time between 2 and 3.

The 25-hour day:

• The bar chart looks wrong. The x-axis does look right (notice that it gets stretched around the DST transition), but one hour of data is erroneously moved to the left. -> I suggest we open a separate bug report for this.
• The matrix chart looks wrong, but it's hard to come up with a better visualization that still lets us see patterns by having local hours be aligned vertically. Especially one that is reasonably implementable. Any idea? Stretching the x-axis doesn't work well for this data visualization, because it would introduce a mostly empty column between 2 and 3. My best idea so far would be to split up that one hour vertically (which sounds kinda hard to implement). Depending on the effort, we may also consider addressing this in a separate issue.

[victorgarcia98]

The matrix chart looks wrong, but it's hard to come up with a better visualization that still lets us see patterns by having local hours be aligned vertically. Especially one that is reasonably implementable. Any idea? Stretching the x-axis doesn't work well for this data visualization, because it would introduce a mostly empty column between 2 and 3. My best idea so far would be to split up that one hour vertically (which sounds kinda hard to implement). Depending on the effort, we may also consider addressing this in a separate issue.

Indeed, this edge case breaks the whole point of this data viz (align hours).

I think the most important is to let the user know that they are seeing a DST time change and not a weird bug that might be affecting the data. For that, It would be great to show a mark indicating the moment of change of hour.

Also, maybe we could let the user chose if they want to use CET/CEST...

In any case, I think we should leave these improvements for a follow up issue.

[nhoening]

I feel the naming issue is still open. I see "heatmap" and "matrix" being used, and the UI now says "Time of day".

I realize one name might be the one in the code, one the user-facing label. They should nt be completely different though.

Maybe the user label can be "Daily activity heatmap" and we call it "heatmap" internally?

[nhoening]

Does the replay button work well in this chart type?

[Flix6x]

Does the replay button work well in this chart type?

It works well, yes. Not perfect, though. The data shows up perfectly, but the one thing I couldn't get precisely right was the replay ruler. It is full-chart-height rather than the height of the currently replayed day. It was one of those things that seems like a really tiny thing, but then ends up taking hours (to, in this case, as of yet, not succeed).

[nhoening]

The ruler doesn't seem to be worth hours of our time.

[Flix6x]

• I incorporated the word daily into the new chart type name (daily heatmap), with an eye on supporting a weekly heatmap soon. I already did a tech spike.
• I decided against incorporating the word activity into the new name; it didn't seem right while I was testing the new chart on e.g. price data.

[nhoening]

I can't seem to be able to test this feature locally. I'm running the right branch (see log below), I look at a sensor page, but there is no chart selection option:

(fm-3.10) nicolas at pop-os in ~/workspace/seita/flexmeasures on sensor-matrix-chart (6 untracked)
± git pull
Already up to date.
(fm-3.10) nicolas at pop-os in ~/workspace/seita/flexmeasures on sensor-matrix-chart (6 untracked)
± flexmeasures run --no-reload
[FLEXMEASURES][2023-07-28 21:33:10,984] INFO: Loading config from /home/nicolas/.flexmeasures.cfg ...
Warning: Without complete mail settings, FlexMeasures will not be able to send mails to users, e.g. for password resets!
You might consider setting MAIL_USERNAME, MAIL_PASSWORD.
[FLEXMEASURES][2023-07-28 21:33:11,686] INFO: Importing plugin flexmeasures_openweathermap ...
[FLEXMEASURES][2023-07-28 21:33:11,713] ERROR: Missing config setting 'OPENWEATHERMAP_API_KEY' (You can generate this token after you made an account at OpenWeatherMap.).
[FLEXMEASURES][2023-07-28 21:33:11,713] INFO: Loaded plugins: {'flexmeasures_openweathermap': '0.1'}
 * Serving Flask app 'flexmeasures.app:create' (lazy loading)
 * Environment: development
 * Debug mode: on
[FLEXMEASURES][2023-07-28 21:33:11,728] INFO:  * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

Am I missing something?

[Flix6x]

Perhaps a hard refresh in the browser? Can you check the page source for the chat type button?

[nhoening]

I see it now, not sure what went wrong.

But I can't test this feature because I cannot upgrade my database with this step:

d814c0688ae0 -> 2ac7fb39ce0c (head), add attribute column to data source

That's no part of this PR.

[nhoening]

I could resolve this local issue and now I can test. It works.

Some limitations are that sources seem to be not labelled or not distinguished? See my screenshot above, it has two sources for the same sensor.

I'm not sure if they are mixed or so. Also, one source gives schedules (Seita), the other measurements (Ard), I believe. The playback results in "interesting" behavior.

I can make a screencast if needed.

[Flix6x]

Good catch. Maybe I can create something to toggle between sources, because showing multiple sources in a single heatmap won't make for a very useful visual imo.

Unless the sources don't overlap in time (e.g. when one source takes over from another). In that case it might make sense to show only the most recent source for each time step, and possibly indicate the transitions.

Perhaps the second idea can be a follow-up issue, and for now we either support toggling or simply raise if the selected time period contains multiple sources.

[nhoening]

I'm for toggling, but it seems that might also be hours of work.

Isn't toggling by source a feature we would like to have in general? Then that could be done with a benefit to the whole UI, and with this type of chart we should allow only one to be active at a time.

[Flix6x]

I went some quite some way into getting the toggle selection to work, but in the end I abandoned it, because I can't manage to initialize the selection to show 1 source instead of none (or all) of the sources, at least not within vega-lite. So for now I decided to show the source with the most data (in the chart).

[nhoening]

Alright, so now we only show data from one source, with no way to adapt.
I know this is still filed under "experimental", but should we not mention this below the chart (maybe if there are more than one source, or just in genera)?

[nhoening]

Otherwise, I confirmed locally that it works :)