Alerting

Customize time for alerts

theouhuios
Motivator

Hello

I have a alert running for every 5 mins to check the service status and alert when it's STOPPED. Everyday from 3:15 to 4:15 all the services will be STOPPED. Is there anyway that I can make that search not to run for that 1 hour and stop sending emails. As we all know that services will be down for sure in that 1 hour period we don't need to get alerted for that 1 hour.

Any ideas

Tags (1)
0 Karma

BobM
Builder

Hi Theo

If your server was down for a clean hour, it would be a simple task to use cron to schedule it. For example the following would work every 5 mins from midnight to 2:55 and then 4:00 onwards.

*/5 0-2,4-24 * * *

For your timing, this is not so simple. I would normally use the date_hour and date_minute fields to calculate the time but your report may not have them. What you could do is modify the search by adding the following at the end of the search.

| eval timeInMin=(now()-relative_time(now(), "@d"))/60 | where (timeInMin < 3*60+14) OR (timeInMin > 4*60+16)| fields - timeInMin

This would delete any results if the time was between 3:15 and 4:15. If this doesn't work, please tell us how you have your alert set up.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...