[Feature Request] Allow Template Variables To Be Used in Alert Queries

[anybodysguest]

Presently, you cannot use template variables in alert queries. Not having the ability to use template variables is a major gap in functionality since we have to hard code values in our queries to avoid using template variables. For example, many of my queries reference a template variable named "environment" (among others). This means that a perfectly good query I have which references the template variable cannot be used when defining alert. To make the alert work I have to remove any template variable references in the alert query and hardcode the value; thus I can no longer use the benefits of templating. This adds up to a lot of work and doesn't allow me to take advantage of templating. Templating is essentially useless when using alerts.

At present count, I have over 20 alerts with more to be added. Every one of these queries have to be scrubbed to ensure that they aren't using template variables.

Can we please get this fixed so that template variables can be use in alerts queries? I'm sure this is causing a lot of users pain.

I was hoping 8x would have solved this problem but it still exists in 8.1.2 (the version I'm using).

I've seen several threads talking about how to "work around" this issue going back several years and it's still a problem. I don't understand why Grafana Labs is fielding these requests to allow template variables in alert queries yet it still persists. To me this is major functionality which is broken and should be fixed - without workarounds. You will continue getting these requests to have it fixed by others because it is counter-intuitive that an alert (which presumably works off the result of the data the query is returning) won't work when a template variable is present in the query.

PLEASE FIX!

[Meruem94]

This feature need to implémented.

[Ultraseamus]

I'm sure the main complexity here is what to do with that templated value. Do you try to run to alert for all possible combinations of the template variables? Probably not, since that could get out of control quickly. Or you go with whatever the defaults for that dashboard are? Which seems too restrictive to be useful.

My suggestion would be to include part of the alert configuration that allows you to set values for each Template Variable present in your alert (which would end up being carried around similar to the "ScopedVars" object). Then you can configure alerts on panels that use Template Variables, and the only real complexity is adding the UI elements to select the values to use for the alert.

As a v2 further down the line, it could be given a sort of "Repeated Panel" option where you are specifically asking Grafana to iterate through all values for a single Template Variable. If you wanted to get fancy, those "Repeated Alerts" could even just fire a single email.

[tom0010]

I had the same problem, and I figured out it was because I was using a "classic condition" when building the query. You are right, you don't have access to them variables that way.
If you use "reduce" or "math" then they become accessible.
The docs does say this:

Name	Description
$labels	The labels from the query or condition. For example, {{ $labels.instance }} and {{ $labels.job }}. This is unavailable when the rule uses a classic condition.
$values	The values of all reduce and math expressions that were evaluated for this alert rule. For example, {{ $values.A }}, {{ $values.A.Labels }} and {{ $values.A.Value }} where A is the refID of the expression. This is unavailable when the rule uses a classic condition.
$value	The value string of the alert instance. For example, [ var='A' labels={instance=foo} value=10 ].

That was my workaround for this. I've been using Reduce, min, a and then another query, Math, $B < 6 for example.
I ended up using a webhook in the end and built my alert from the payload using a slack bot, anyway, hopefully this helps.

[anzecesar]

My 2 cents: We have a dashboard with the overview of the multi tenant system. The variable is the id of each tenant.

We need alerts on certain metrics, per tenant.

Without alerts supporting variables, we need to run our own inhouse hacks, which is not fun.

We have more than 1000 tenants, to making a dashboard for each one is not really feasible.

[andrewgkew]

Is there any update on this discussion? Feature request in progress @grafanaTeam

[wardbekker]

Hi folks, check https://grafana.com/docs/grafana/latest/alerting/unified-alerting/ for the new, more powerful, Grafana alerting.

[grobinson-grafana]

Hi,

I think this use case is now supported in Grafana 8 (please see multi-dimensional alerting).

If we use the first example where we have a dashboard with a template variable env, and this template variable matches the env label, with multi-dimensional alerting Grafana will create one alert per series for each series that should create an alert.

For example, if we have two envs: dev and prod, a job called foo in each env, and the following alert rule:

up{job="foo"} == 0

then, if foo was down in both dev and prod we would have two alerts with the following labels:

{env="prod", job="foo"}
{env="dev", job="foo"}

If we want to create an alert for prod and staging, but not dev, we could change the alert rule to something like:

up{env=~"staging|prod",job="foo"} == 0

[gotjosh]

Hey all,

As @wardbekker mentioned, this is now possible with https://grafana.com/docs/grafana/latest/alerting/unified-alerting/ - I'm going to close this issue now, but I'd encourage whoever is reading to open another issue if they find something is missing.