Alerting: Time of day restrictions

[torkelo]

Time of day restrictions.

See two ways this could be implemented.

1. As an alert condition
2. As a filter on notifications

[bblazei]

Has anyone come up with a workaround for this scenario? I'm shocked that only 4 people have upvoted this ticket as the entire alerting feature is essentially rendered useless for me unless my systems are up 24/7. I feel like I must be missing some other feature or technique that everyone else is doing to circumvent this problem...

As it stands now, I either have to turn all alerting notifications off , or just accept the fact that I will get a bunch of spurious notifications when my processes shutdown EOD.

Not trying to sound critical really, just confused how everyone else is managing these alerts. I've loved Grafana for years now, and I've been tracking the alerting feature since it was first introduced in v4. But it's a bit of a head-scratcher every time there's an alerting update and this limitation isn't addressed.

[JulienChampseix]

@bblazei you're right! it's a amazing feature which need to be priorised and for sure which will be helpful ppl for sure !
@torkelo do you know when this feature will be planned ?

[torkelo]

No not Eta right now as it's not on our roadmap for the next two releases (4.3 and 4.4)

[bblazei]

Hmm well that's unfortunate. How would you recommend using the Alerting framework on systems that are not 24/7?

[joelraburton]

This is something that we would very much like as we need to have different alert levels depending on the time of day

[cjs226span]

We are (not so) patiently waiting for this too. We currently utilize curl to Slack graphs periodically.

[rajpaly]

@torkelo torkelo. its been a while i dont see an update on this. we are also looking to some thing like if we can disable the grafana alerts for specific time. is that possible?

[yacut]

Has anybody some update for this feature?

[stevebanik-ndsc]

I'm able to manually pause alerts on the Alert List page, but (for example) during our daily 2:30a DB server backup we get an alert on "Network I/O Waits In Progress." It would certainly be nice to create alerts that it do not notify during certain periods of time.

[fsobanski]

Does grafana support a modulo operation? Then, you should be able to use the identity function to get the unix time as an additional metric in your panel. With the modulo function you could get the remainder of the division of the unix time by 86400 (the number of seconds in a day). Then you could add a range condition on the time metric in your alert. Right?

Would it be difficult to add the modulo operation for this purpose?

[solosky]

Really need this feature!

[cherryghost]

Any updates on this? Is it a WIP or something still just being 'considered' right now?

[jsimpsoncd]

We really would like to use different timeframes for certain alerts like usage, which we expect to be above a certain threshold during the day but not at night.

[pete-leese]

+1

[lilaboc]

+1

[julienlefur]

+1

[matt-conrad-ck]

Why are people (@bascarsija & @maizy) down-voting people's requests for this?

[bascarsija]

people are "down-voting" these "+1" messages that cause email notifications to be sent to everyone who is subscribed to this thread. the cumulative effect of many people adding these "+1" messages to the threads significantly reduces the value of the thread subscription feature by dramatically reducing the signal-to-noise ratio.

you can indicate your interest in or agreement with specific proposals or comments made by others without triggering any such email notifications by simply "up-voting" or "down-voting" via the reactions. the most prominent display of solidarity/aggregate interest in an issue is typically the reaction counts on the initial/primary description of the issue -- please consider adding your reactions there.

frankly, this is a systemic problem with Github -- it applies equally to all issue threads in all projects. you will find many pleas across the various threads requesting such usage because the Github UI does not inform users of this consequence or discourage it in obvious any way, and users who are aware of the problem (and/or are negatively impacted by it) are likely hesitant to provide such feedback by adding yet another message to the thread (thereby decreasing the signal-to-noise ratio further).

[matt-conrad-ck]

Thanks for the explanation. I'm guessing the people posting those +1s wouldn't have done it if they knew it was just an annoyance. Posting a short explanation about that might have stopped a few of those... as well as my question. Thumbs-down could mean a number of things.

[nkshah]

Any update on schedule alert(s) during the certain time of the day, week, month and year?

[albertvaka]

As a workaround, using prometheus as backend:

• Add the following query to your metric: hour(), that returns the hour of the day (0-23). You can make it hidden in the graph.
• Add an extra AND condition to your alert, so it's only alerting if the hour() query is between the range you want (eg: office hours).

The same can be done with day_of_week().

[Waynedejager1]

@torkelo - would you mind letting me know if this feature is happening in 8.1.x?

[gotjosh]

We're aiming to have it by 8.2 but no later than 8.3.

[Waynedejager1]

@gotjosh fantastic ! Thank You

[bblazei]

Hey all, checking back in. Looks like some of you were able to check out my fork, and hopefully it helped you guys out!

Anyway, I noticed that 8.1 came out a few weeks back and still no alerting changes... womp womp. But josh has reassured us it will be out in one of the next releases. Which probably means it's actively being worked on right now. But we also don't know the timeline on when 8.2 or 8.3 will be rolled out...

So, in the meantime, did anyone have any feedback on my changes? If there's demand, I'm happy to make adjustments or try porting to other previous releases.

Let me know!

(#33075)

[chrisfw]

Hi, A coworker pointed out that there is a new feature in version 8 that allows you to schedule a window to silence alerts. I have not investigated it yet so I’m not sure that this will work for time of day restriction, but it is a step in the right direction.

…

On Sep 22, 2021, at 11:05 AM, bblazei ***@***.***> wrote:  Hey all, checking back in. Looks like some of you were able to check out my fork, and hopefully it helped you guys out! Anyway, I noticed that 8.1 came out a few weeks back and still no alerting changes... womp womp. But josh has reassured us it will be out in one of the next releases. Which probably means it's actively being worked on right now. But we also don't know the timeline on when 8.2 or 8.3 will be rolled out... So, in the meantime, did anyone have any feedback on my changes? If there's demand, I'm happy to make adjustments or try porting to other previous releases. Let me know! (#33075) — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[fifeclub]

We're aiming to have it by 8.2 but no later than 8.3.

It doesn't appear to have made it into 8.2.
Any info on 8.3 timeframe?

[gotjosh]

There was a slight change of priorities last minute for 8.2, but we're all systems go for this to be included in 8.3.

[tyagiakhilesh]

Thanks. Eagerly awaiting for this. I figured that its been going for so many years, that time for someone to pull up the sleeves ourselves. But looks like its planned in 8.3 now ? I don't see milestone tagged to issue. Is it really planned ?

[devopsusr-tech]

@gotjosh was happy to hear that grafana released version 8.3 today, but this feature seems not to be implemented yet :(

[agido-malter]

Sorry, but i think it will never released. This is a request from 2016 and we are now in end of 2021

[perplexityjeff]

So many years but there is some light at the end of the tunnel. It does get pushed back a lot.

Check here #41578

[gotjosh]

Thanks everyone for your patience. As @perplexityjeff pointed out, the change is already under review and we're aiming to include it as part of 8.3.x.

Open-source software is tricky, and for this one, we wanted to make sure we re-used components for other OSS projects so that we can continue to foster this big community together.

We're almost there 🤞

[pkkrusty]

Fix for this was merged into main early Jan 2022. So we really are almost there.

[skob]

hi,
any newz here?

[brunobastosg]

Is this the new Silence feature? If so, is there a way to use ONLY a time range? I want to silence all alerts overnight.

As seen in the screenshot below, I'm forced to select a date.

EDIT: just found about #37752 which is exactly what I need

[M0rdecay]

In our case, we are using grafana as a simple alerts source, but for some complicated cases, like auto silences in time intervals, we have alertManager - https://prometheus.io/docs/alerting/latest/configuration/#mute_time_interval

I hope, someone finds this useful

[pkkrusty]

https://grafana.com/docs/grafana/latest/alerting/unified-alerting/silences/

Yeah, that's pretty weak, to only have a one-time silence. I would imagine that 95% of use-cases are going to be regular, recurring silence periods.

I guess I can't knock it too much, since someone took the time to program this feature for free, but I'll put my vote in for allowing recurring time range.

[Waynedejager1]

https://grafana.com/docs/grafana/latest/alerting/unified-alerting/silences/

Yeah, that's pretty weak, to only have a one-time silence. I would imagine that 95% of use-cases are going to be regular, recurring silence periods.

Couldnt agree more - its very weak to only have a once off silence. A huge deviation from the request a few hundred of us have been waiting on since Nov 2016.

@gotjosh Will the original consideration still be included down the line or is the watered down attempt a replacement ?

[gotjosh]

That's the documentation for silences. The documentation for mute timings is https://grafana.com/docs/grafana/next/alerting/unified-alerting/notifications/mute-timings/

As a final note, this is now merged and released under 8.4 (as opposed to 8.3 as we originally aimed). The first beta for 8.4 is out so give it a try when you can!

[Waynedejager1]

@gotjosh Champion!!

[gillesdemey]

Closing this one; mute timings will ship in the next release of Grafana, 8.4.0.

[dglushenok]

Documentation is here, actually: https://grafana.com/docs/grafana/next/alerting/notifications/mute-timings/

[lfkdev]

both mentioned mute-timing links are dead:
https://grafana.com/docs/grafana/next/alerting/unified-alerting/notifications/mute-timings/
https://grafana.com/docs/grafana/next/alerting/notifications/mute-timings/

correct link:
https://grafana.com/docs/grafana/latest/alerting/notifications/mute-timings/

[olahellgren]

Good to know for you who end up in this thread. You can also use hour(), day_of_week() as queries use them in your condition for the alarm. This way the alarm can be active only at specific times.

[chrisfw]

@olahellgren If I'm not mistaken one caveat with hour is it returns the value in UTC, so you need to adjust for the current offset if you deal with daylight savings time or your alert condition may not be accurate.

[olahellgren]

@olahellgren If I'm not mistaken one caveat with hour is it returns the value in UTC, so you need to adjust for the current offset if you deal with daylight savings time or your alert condition may not be accurate.

Good input. That's good enough for me.

[gtirloni]

I need to mute an alert (or the notification), on specific days and times. I cannot. I can only mute all notifications globally as I don't see any form of filtering with labels in the mute timing settings.

[gillesdemey]

Hi @gtirloni! Mute timings are not global or don't apply to all alert rules – you assign them to a notification policy.

Being able to specify label matchers for mute timings is currently not supported by Alertmanager but that sounds like a reasonable feature request – please create a new issue so we can follow-up on that so we don't have to keep bumping this issue.