Long term statistics support

[dbuezas]

Thanks to @FrnchFrgg , this version brings support for statistics!

type: custom:plotly-graph-dev
entities:
  - entity: sensor.temperature
    statistic: mean # min, mean, max or sum
    period: hour # 5minute, hour, day or month

This means that multi day plots can now be very snappy, and you can look way back in the past too.

After we added statistics support, I decided to make a fairly big refactor to improve the API of both statistics and attributes.
Because of that, this one is marked as beta version. Please feel encouraged to report any bugs as issues, or drop a thanks to @FrnchFrgg in the discussions section if you were missing the feature!

---

This discussion was created from the release Long term statistics support.

[netweaver1970]

thanks @FrnchFrgg and @dbuezas ! How do I install this beta version next to the normal version?

[dbuezas]

It's quick and simple, see here:
https://hacs.xyz/docs/faq/beta/

[dbuezas]

Out of beta in the next 5 minutes

[Zingel1986]

Hi, thanks for the good work. I wanted this feature since the first version.
Is it possible to change the statistic period when changing the view with the range selector?

[dbuezas]

I exaggerated the effect a lot so it is visible (set the resolution to a minimum of 25 datapoints in screen)

[dbuezas]

And here is with a minimum of 500 datapoints. The difference between hour and day is because I'm plotting temperatures (which obviously have a cycle of exactly 1 day).
But between 5 minutes and 1 hour, it looks great

[dbuezas]

Experimental branch:
https://github.com/dbuezas/lovelace-plotly-graph-card/tree/feat/statistics-automatic-period
It shows the selected period in the title
It's a bit of a hack, but it does seem to work.
Feedback appreciated.

[Qhilm]

Is the behavior of period: auto documented somewhere?

[dbuezas]

Yes, in the readme https://github.com/dbuezas/lovelace-plotly-graph-card?tab=readme-ov-file#automatic-period

[dbuezas]

Here is a gorgeous example from @r-jean-pierre
Do you mind posting your yaml Jean Pierre?

[r-jean-pierre]

Here is where I am so far

cards:
  - type: custom:plotly-graph  # Kid's room
    defaults: &mon_defaults
      entity:
        texttemplate: "%{y:.1f}"
        connectgaps: false
        hovertemplate: ''
        line:
          shape: spline
          width: 2
    entities:
    - entity: sensor.ths_kids_temperature
      name: T[°C] (max)
      line:
        color: rgba(81, 117, 159, 0.4)
        width: 1
      statistic: max
      period: 1day
      showlegend: false
      show_value: false
    - entity: sensor.ths_kids_temperature
      name: T[°C] (min)
      fill: tonexty
      fillcolor : rgba(81, 117, 159, 0.2)
      line:
        color: rgba(81, 117, 159, 0.4)
        width: 1
      statistic: min
      period: 1day
      showlegend: false
      show_value: false
    - entity: sensor.ths_kids_temperature
      texttemplate: "%{y:.1f}°C"
      name: T[°C] (mean)
      line:
        color: rgba(81, 117, 159, 1.0)
      statistic: mean
      period: 4hour
      show_value: true
    
    - entity: sensor.ths_kids_humidity
      name: φ[%] (max)
      line:
        color: rgba(143, 86, 164, 0.4)
        width: 1
      statistic: max
      period: 1day
      showlegend: false
      show_value: false
    - entity: sensor.ths_kids_humidity
      name: φ[%] (min)
      fill: tonexty
      fillcolor : rgba(143, 86, 164, 0.2)
      line:
        color: rgba(143, 86, 164, 0.4)
        width: 1
      statistic: min
      period: 1day
      showlegend: false
      show_value: false
    - entity: sensor.ths_kids_humidity
      texttemplate: "%{y:.1f}%"
      name: φ[%] (mean)
      line:
        color: rgba(143, 86, 164, 1.0)
      statistic: mean
      period: 4hour
      show_value: true
    
    layout: &mon_layout
      hovermode: "x unified"
      yaxis:
        range:
          - 10
          - 40
        gridcolor: rgba(233, 233, 233, 0.4)
        zeroline: false
        showticklabels: true
        tickfont:
          color: rgba(81, 117, 159, 1.0)
        title: false
      yaxis2:
        range:
          - 40
          - 90
        gridcolor: rgba(233, 233, 233, 0.4)
        zeroline: false
        tickfont:
          color: rgba(143, 86, 164, 1.0)
        title: false

      margin:
        "l": 32
        "r": 32
        "t": 64
        "b": 40

      xaxis:
        gridcolor: rgba(233, 233, 233, 0.4)
        rangeselector:
          "y": 1.12
          "x": 0.72
          buttons:
            - count: 12
              step: hour
            - count: 5
              step: day
            - count: 30
              step: day
    hours_to_show: 72
    refresh_interval: 30
    title: Kid's room

[dbuezas]

Did you see this?

https://www.home-assistant.io/images/blog/2022-11/long-term-stats-more-info-dialog.png

[netweaver1970]

I saw it too, I'm on an early beta, since I saw the water usage metering and the LTS/week support. Both are something I wanted.
But they just went to LTS instead of real state values. Lots of discussion on this change on the Beta Discord channel on the (un)usability of this change ...
For me it's not enough, I like the flexibility, the auto-ranging, zooming etc from Plotly, if all works out as planned/hoped.
Their (HA team) reason was just to reduce the cpu/system load by using the LTS instead of state values.

[Qhilm]

@r-jean-pierre , I am unable to use period: 1minute. I also tested "1day" like in your yaml example. The error I get says:

period: "1minute" is not valid. Use 5minute,hour,day,week,month

Do I need to define custom periods somewhere else?

[dbuezas]

Were those added in home assistant's statistics API?

[r-jean-pierre]

I think it's the 5 periods we can select with the combination statistic/period:

and true that it's not possible to tweak it

Now I look at my own example, it shouldn't work, and currently this is not what I have in my running Yaml.
I switched to this technique:

I don't remember why it did it, more powerful or had technical issue (?)...

[Qhilm]

thanks.

I don't fully understand the new syntax: is this dynamic depending on the time windows displayed on the chart?

And secondly, what does "0s" mean in this case?

[dbuezas]

The left side is the minimum visible range, see readme

[r-jean-pierre]

Hi!

I'm posting here cause the title makes sense ("Long term statistics support")

I have a simple and easy graph with 5mins average

      - type: custom:plotly-graph
        entities:
        - entity: "sensor.waterheater_temperature"
          statistic: mean
          period: 5minute
          showlegend: false
          line:
            width: 3
            shape: spline
            smoothing: 0.5
          show_value: true

If you look on bottom left corner (red), I tried to display 2 weeks, but data history length seems to stop ~10 days ago
It matches (?) with the source "History" of HA , but there is data in the source "Long Term Statistics" (as you can see on the right side of the screenshot in blue)

I just noticed "recently", when I was forced to restore a backup to restart my HA setup and therefore lost some of my history (it was two months ago)
Before, I was able to zoom out indefinitely, thus display months of data (I suppose I was using without knowing some internal cache/history from your integration or something else (?) ) but since, I had time to accumulate data and only the last 10 days are displayed.

It seems to have the same behaviour without statistic: mean period: 5minute

Is there something I missed (or misuse) ?

[dbuezas]

Hehe. Using plotly in a notebook?

[r-jean-pierre]

Of course ! +seaborn when I don't need interactivity. Finally, my issue is not the graph rendering library-ies, but data preparation... Python+pandas come to the rescue there. And I would say even much most straightforward: using functions! and constants! My YAMLs are ugly. A real HA-ninja-expert would probably say "it's possible", but out-of the-box, with only 5 neurons: it's impossible!!! Last but not least: even with custom:layout-card from thomasloven, I'm still struggling in some situations with my card layout, border, margin, padding etc. and sometimes have to give up on tasks because they seem impossible to accomplish

[dbuezas]

It would be nuts if there was a jupyter notebook card

[r-jean-pierre]

Yes indeed, and talking about data+maths (because I always focus on pixels with plotly!), I have a question/doubt:

Assuming I want to use statistic with aggregation hour, what is really computed?
When the aggregation put a timestamp at 15:00 is the average computed on range [14:00, 15:00] or [15:00, 16:00]? Or even better: [14:30 15:30]?

There is something that I don't get and looks shifted of almost 2H:

Top sensor is not aggregated, I already got data after 17:00, if mean that from 16:00 to 17:00 we can compute something, but the earliest aggregated data is at 15:00 (Bottom graph is min/max/mean per hour of the same sensor)
It seems from pixel point of view that the aggregated value at time 15:00 is probably the aggregation from 14:00 to 15:00 (which replies to my first question, but only based on 1 example)

But why there is this shift in time? Computation is probably right, but it occurs too late

Last but not least, as a challenge, have you ever consider to compute and display the last "unfinished" bloc of 1H? I mean, assuming it's 15:20, why not computing avg over 15:00 to 15:20 and display the current avg at time 15:20. Of course it makes more sense and more useful at 15:59!!! Could be a great option as there is no risk with min, mean of max functions even with one data point

[dbuezas]

I think home assistant does what you mention: at the end of the period, compute stats from last period (e.g 1h) & store them. And whatever HA stores in the db, this card plots.
Regarding the delay, I don't know tbh. You could test it using $fn :)

To compute the unfinished part you may find this discussion interesting