Y-axis scale by percentile of data values

[rvdheij]

What would you like to be added:

An option to have the Y-axis automatically scaled so a percentile of the data points, say 95%, fits on the chart. This would probably be a separate setting for both min and max of the scale.

Why is this needed:

This would allow the chart automatically show most of the data with good granularity such that a single brief peak does not turn the entire chart into "grass" when raising the Y-max. In most cases it is enough to see that a short peak was off-scale, and the actual value is rarely interesting (as it depends on when the spike exactly was, and whether it spans one or more intervals). You could still hover over the chart to see the actual value, or zoom in the time range so that the peak is most of the data and the Y-max is adjusted to capture it. This is also helpful when a derivative() is used and the first reading is off-scale.

You could achieve the same by setting a hard Y-max, but that defeats the purpose of the template variables to use the same dashboards for different systems (that have different typical good values).

[leeoniya]

i think what you're asking for is covered by softMin/softMax?: #30326

[rvdheij]

@leeoniya I don't think so, though I was certainly inspired by that change. I seem to leave a trail of wanting the opposite :-)
As I understand softMax, it is the bottom of the range in which the Y-max is chosen. That prevents minimal variations in a value to be magnified by the scale as if it were all over the place.
Imagine the case where the value varies between 100 and 200 and these variations are relevant to see. When there is a sudden short spike of 1000, the Y-max is adjusted to catch the 1000 and hides any subtle variations before and after the spike. I was looking for something that ignores a few points off-scale. I suppose it would have the side effect of keeping the Y-max more steady and possibly keeping different charts in synch when the median is in the same range.

[rvdheij]

Playing with the new Time Series panel in 7.5, I notice that setting the Y-axis unit to "Percent (0-100)" is also setting the maximum to a hard 100. That's different from the Chart panel and not desirable. I don't see that I can undo that with another setting. I have a lot of cases where percentage can go above 100 (but want the values to say "%") and also several periods where we don't reach the 100% (especially when you're looking at one of the series in isolation). Should I open a separate issue to encourage you to keep those settings apart?

[leeoniya]

Playing with the new Time Series panel in 7.5, I notice that setting the Y-axis unit to "Percent (0-100)" is also setting the maximum to a hard 100.

by % may not be fully baked yet. there's still more to do: #30564

When there is a sudden short spike of 1000, the Y-max is adjusted to catch the 1000 and hides any subtle variations before and after the spike.

this case is covered by setting both the softMax and the hardMax.

try importing this dashboard to see it in action: https://github.com/grafana/grafana/blob/master/devenv/dev-dashboards/panel-graph/graph-ng-soft-limits.json

I suppose it would have the side effect of keeping the Y-max more steady and possibly keeping different charts in synch when the median is in the same range.

yes, that is a side-effect and generally desirable, too.

[rvdheij]

I will play a bit more with the new panel.

this case is covered by setting both the softMax and the hardMax.

But I don't want to set the maximum so my dashboards can be used for systems with different value ranges, and also for when you select one of the series to study in isolation.

[leeoniya]

as with everything, it's a trade-off.

i guess you're asking to clip outliers based on some % of a stat of the in-view data (like mean or geometric mean) but still auto-scale the Y dynamically based on the non-outlier range (which would make Y less stable, but more adaptive between different datasets).

i can see this being useful, but not sure how much appetite there will be for additional settings in the UI. the softMin/softMax took a good amount of convincing to get the real-estate approved. we can leave this open to see how many upvotes it gets in a few months.

[rvdheij]

It helps when I finally managed to explain what I was suggesting. I don't think it has to take more real estate as you could have just a value plus an indicator whether that value is hard, soft, or computed percentile of the data. I think having separate soft and hard values is a bit too much.

[leeoniya]

I think having separate soft and hard values is a bit too much.

it's actually absolutely necessary. you can read the entire, very long thread, but specifically: #979 (comment)

[rvdheij]

Thanks. And before I forget to say it, I am very pleased with what got into the new Time Series panel.
It would be interesting to see what my approach would do to the scale when you have a sliding window over it...

[pkolyvas]

We're in the process of moving Grafana feature requests to GitHub Discussions. This issue is being moved to Discussions or has been moved from our issues list, depending on where you're seeing this message.

We hope to use the discussion thread to continue building use cases, and additional evidence based on engagement/voting, in order to help us prioritize this feature request later on down the road.

As with all things Grafana, we value your input and participation in this project and appreciate your patience during this transitional period. Looking forward to the discussion!