Include alert threshold in plotted range

[Krinkle]

• Grafana version: v5.1.3 at https://play.grafana.org, and v4.6.3 at Wikimedia.

Problem

When creating an alert for a graph, by default there is no way to find out what the threshold is by viewing the dashboard. Below is a modified example from https://play.grafana.org/d/000000074/alerting?panelId=1&orgId=1. I will explain why it is modified.

The graph indicates positive health of the alert (green heart) but the threshold cannot be seen.

The interesting part is that Grafana does have the ability to plot the threshold. The problem is that it only shows the threshold in the plot, if the plot always reaches the value in question. As shown above, if the plot does not reach the threshold, then the line is not visible.

Request

Please allow the graph to consider the alert threshold as one of the metrics so that it will always render the graph at the correct height so that the threshold line is visible.

Alternatively, provide other ways of viewing the threshold in the graph. For example, in the legend, or somewhere else that can be always visible.

Workaround 1: Manual Y-max

Even the Grafana demo recognises this limitation. When the metric Y-axis does not reach to the threshold, the line is not visible. The Grafana demo works around this limitation by artificially setting the Left Y-max to "40" for the above case, making it look like this:

Simple alert	Manually set Y-max

This workaround is nice for demonstration purposes, but not useful in production because it will cut off the graph, because it sets a maximum for the ceiling of the plot, instead of a minimum.

Below is an example of this going wrong.

Metric violates threshold	Problematic Y-max

Workaround 2: Manual threshold metric

Another workaround, is the one we use at Wikimedia. Namely, by manually creating an extra metric in the plot, using a query that represents a constant value. As example, for Prometheus this would be a bare numeral value. And for Graphite this could use constantLine().

Then, in the Series override (Display), one has to disable "Line fill" and "Legend".

Step 0: Graph with alert and unknown threshold

Step 1: Add "threshold" metric	Step 2: Add series overrides

This second work around has no problems with it, except that it requires keeping in sync with the alert query. But at least it will work no matter what happens to the data.

[torkelo]

Great issue description, however there already is one for this issue #12135

[Krinkle]

@torkelo Eh, did you mean another number? Because that number is literally this issue.

[marefr]

Maybe #5984 was intended?

[torkelo]

yes, sorry, pasted wrong number