ngalert in Grafana 8.0.4: alert gives NoData while there is data (Prometheus node-exporter 0.15)

[AronVanAmmers]

I'm testing the new ngalert feature, very happy this has now arrived after following issue #7832 for a long time :). Experiencing an issue with a preexisting alert, imported from the old alert system.

What happened: we have disk space alerts for data sources based on Prometheus node-exporter. We have some machines still on node-exporter v0.15, some on the newer v0.18. As far as I can see the metrics used haven't changed between these versions of node-exporter, and we're using identical queries for v0.15 and v0.18. All of the alerts work fine in the old alerts system.

However in ngalert, the v0.15 source results in NoData (while there clearly is data visible), whereas the v0.18 works fine.

What you expected to happen: both to work fine.

How to reproduce it (as minimally and precisely as possible):

Use the following query (A): 100 - ((node_filesystem_avail{device!~'rootfs'} * 100) / node_filesystem_size{device!~'rootfs'})

With a condition "WHEN avg() OF A IS ABOVE 60".

Output with a Prometheus node-exporter 0.18 data source (as expected):

Note the "0" under Value, that's good. Further down we see the state of the alert is Normal:

Output with a Prometheus node-exporter 0.15 data source, after pressing Run queries (shows the issue):

Note the empty space under Value. Further down, it's clearer this results in NoData:

Anything else we need to know?:

Environment:

• Grafana version: 8.0.4
• Data source type & version: Prometheus 2.1.0, prometheus-node-exporter 0.15.2 and 0.18.1
• OS Grafana is installed on: Ubuntu 18.04
• User OS & Browser: Windows 10, Brave Browser
• Grafana plugins: none
• Others:

[AronVanAmmers]

I've done a bit more analysis. Our query is:

100 - ((node_filesystem_avail{device!~'rootfs'} * 100) / node_filesystem_size{device!~'rootfs'})

If I remove either part of the division, it works well, so doesn't result in NoData. So:

100 - (node_filesystem_avail{device!~'rootfs'} * 100)

and

100 - (node_filesystem_size{device!~'rootfs'})

Both result in the condition evaluating to a number (1 in this case), not NoData.

[AronVanAmmers]

Other math operators also result in an output value correctly, not in NoData:

100 - ((node_filesystem_avail{device!~'rootfs'} * 100) - node_filesystem_size{device!~'rootfs'})

100 - ((node_filesystem_avail{device!~'rootfs'} * 100) + node_filesystem_size{device!~'rootfs'})

100 - ((node_filesystem_avail{device!~'rootfs'} * 100) * node_filesystem_size{device!~'rootfs'})

100 - ((node_filesystem_avail{device!~'rootfs'} * 100) ^ node_filesystem_size{device!~'rootfs'})

And the division operator with a constant also works fine:

100 - (100 / node_filesystem_size{device!~'rootfs'})

Yet the situation where we use the division operator with two metrics results in NoData...

100 - ((node_filesystem_avail{device!~'rootfs'} * 100) / node_filesystem_size{device!~'rootfs'})

[AronVanAmmers]

@kylebrandt tagging you as you were asked to file case-by-case issues for the new alerting system in #7832.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last year. It will be closed in 30 days if no further activity occurs. Please feel free to leave a comment if you believe the issue is still relevant. Thank you for your contributions!

[github-actions]

This issue has been automatically closed because it has not had any further activity in the last 30 days. Thank you for your contributions!