Alert configuration does not change the state if we set $__interval variable in the query.

[anoop2503]

What happened:

Configured an alert panel using Prometheus as the datasource. The query is using the variable [$__interval] for rate function. However, the alert configuration does not change the state based on condition/threshold value.

Set the threshold value as 80, so that if the query result reaches above 80, the alert should be triggered. It didn't happen. Replaced the [$__interval] variable with a value [2m] and the alert got triggered correctly.

Now, changed the rate interval with [$__interval] variable again and waited for the query result come down. However, the alert is not cleared even after the result value less than 80 (which is the threshold limit in condition). Replaced the [$__interval] variable with value [2m] and the alert got cleared.

What you expected to happen:
The alert should work correctly when we have [$__interval] variable in the query.

How to reproduce it (as minimally and precisely as possible):

Configure a panel with alert configuration.
Use the rate function in the query and use $__interval variable for rate interval value
Set the threshold value in the alert condition.
Check whether the alert is triggering correctly?

Anything else we need to know?:

• Used Prometheus as datasource.
• Configured 120s as evaluation interval

Environment:

• Grafana version: Grafana v6.7.4
• Data source type & version: prometheus, version 2.15.1
• OS Grafana is installed on: Running as K8S container. Using Grafana base image.
• User OS & Browser: Chrome
• Grafana plugins:
• Others:

[dprokop]

Hello @anoop2503 and thank you for reporting this issue. Could you please provide us with Grafana logs?

[anoop2503]

This is what I am getting when I click "Test Rule":

Object
firing:true
state:""
conditionEvals:" = true"
timeMs:"5.366ms"
error:"tsdb.HandleRequest() error bad_data: 1:135: parse error: missing unit character in duration"
logs:Array[1]
0:Object
message:"Condition[0]: Query"
data:Object

From Grafana Logs:

{"log":"t=2020-07-07T16:05:18+0000 lvl=eror msg=\"Alert Rule Result Error\" logger=alerting.evalContext ruleId=0 name=\"ABC K8S Master 1 - CPU Usage (%) Monitoring Alert\" error=\"tsdb.HandleRequest() error bad_data: 1:135: parse error: missing unit character in duration\" changing state to=keep_state\n","stream":"stdout","time":"2020-07-07T16:05:18.069422547Z"}

[dprokop]

@anoop2503 thanks for this. Issue is there is no support for variables in alert queries (#6557)