Dashboards: Label exclusion not working in Alert List panel or Alerts Rules

[maxwellvarner]

What happened?

I'd like to have multiple Alert List panels that show different alerts based on their labels. The issue is that when the same label is shared among all alerts I can't filter the Alert List panels to not include alerts with an additional label.

All my alerts have a label of SendTo=OpsGenie and some have an additional label of QuietHours. The QuietHours notification polices are nested under the SendTo notification policy so both labels are required to get the proper routing.

I've attempted to use the following in the Alert List panel's Filter logic in the Alert Instance Label section:

• QuietHours=""
• QuietHours=~^$
• QuietHours!~.+

but in each attempt I either get no results; when I expect to see all my alerts that have the SendTo label, or I end up getting the alerts that do have a QuietHours label.

This same behavior occurs when filtering by label in the Grafana Alert Rules dashboard

What did you expect to happen?

I expect the documented logic for excluding labels to exclude things matching those labels defined

Did this work before?

Not sure if this worked before. Haven't tried it in other versions of Grafnana

How do we reproduce it?

1. Create an Alert Rule with two labels SendTo and QuietHours
2. Create an Alert Rule with just the SendTo label
3. Attempt to use the documented exclude options for labels to filter out the alert rules that have a QuietHours label
4. Observe that they do not exclude them ask expected.

Is the bug inside a dashboard panel?

No response

Environment (with versions)?

Grafana: 10.1.2
OS: Hosted on Linux

Grafana platform?

Kubernetes

Datasource(s)?

No response