Alerting

How to set up a scheduled alert to not send another notification email if the search result is the same as the previous alert result within an 8 hour window?

gyarici
Path Finder

Hi all,

I have some difficulties with one of the my alert logic. Here is the small explanation of my alert query:

I am using;

  • Predict function and I need at least 8 hours time window( latest=-8h,earliest=now) for reliable predict function results
  • Alert schedule is every 10 minutes for detection

My problem is coming from this stage. When the system creates an alert according to search, I receive an email which is ok, but I am receiving the same email result for each scheduled period=10 minutes.

In this case, I tried to use the throttle function to put some optimized time to ignore any alert email. Then I missed a critical alert due to throttled time period. (once per search)

Then I tried once per summary and filled the "Per result throttling fields" with my variable. Now I am receiving an alert email per event in a separate alert email according to my variable and all the emails are same result and receiving in every 10 minutes.

This is kind of dilemma for me.

Is there any way to ignore an alert notification email if the search result is as same as the previous result in selected time window=8 hours and every alarm schedule=10 minutes ?

I would like to stop sending the same alert email result again and again. In parallel, I also would like not to miss any new alert events if it exists in 10 minutes schedule period inside 8 hours time window.

Alarm Result example:
_time ALARM_1 ALARM_2 ALARM_COUNT
Wed Jul 1 15:08:00 2015 YES YES 1
Fri Jul 3 08:28:00 2015 YES YES 2

PS: I am using Splunk 6.2.2

I hope it is clear.

Thanks

Gokhan

Tags (3)
1 Solution

gyarici
Path Finder

Hi all,

I finally figured it out the best way for creating such kind of alert. 😉

First ,I divided my search into two part and using join command.

In order to use similar time period within different time period selection , I used _time function with join.

Now , my search looks like;

 search time range: last 10 minutes

 1st search criteria for alert detection | join _time[search earliest:-8h latest:now|2nd search criteria for predict function]

 cronjob: */10 ****

Now it is working ok.

I hope it helps anyone who needs.

PS: You can use bigger time span for the predict command process now, if you wish to have more data for filtering.

Thanks

Gokhan

View solution in original post

gyarici
Path Finder

Hi all,

I finally figured it out the best way for creating such kind of alert. 😉

First ,I divided my search into two part and using join command.

In order to use similar time period within different time period selection , I used _time function with join.

Now , my search looks like;

 search time range: last 10 minutes

 1st search criteria for alert detection | join _time[search earliest:-8h latest:now|2nd search criteria for predict function]

 cronjob: */10 ****

Now it is working ok.

I hope it helps anyone who needs.

PS: You can use bigger time span for the predict command process now, if you wish to have more data for filtering.

Thanks

Gokhan

srisahitya_v
Communicator

why you take earliest 8 hours for every 10 minutes scheduled alert. it take more data to validate(search) in every 10 minutes.

  1. you can change this earliest time to 10 minutes. or
  2. you can change schedule alert for every 8 hours.

otherwise you will get duplicate results in alerts.

for example: if your alert condition was triggered just now. then you would get same result in next 47 alerts.

acharlieh
Influencer

-1, The asker stated quite clearly the need to have an alert every 10 minutes, and that alert is based on the predict command attempting to predict future trends based on past data. If they do not pick up enough data to feed into the prediction algorithm, the prediction will vary wildly.

vinitatsky
Communicator

We also doing something similar --
We have some alerts (latest=-8h,earliest=now) and we don't want to get alerts again within next 24 hours for same IP Address.

Please find attached image for configurations.
Note - In our case its src_ip is a field.

I hope it helps.

0 Karma

gyarici
Path Finder

Thanks for the reply. I did try this before. My issue is ;

Imagine you have events called Event1 ,Event2,Event3 for IP1 in Time1,Time2 and Time3.(in 24hs period)

Time Event IP
Time1 Event1 IP1
Time2 Event2 IP1
Time3 Event3 IP1

What I need is I would like to receive 3 alert emails as a quick notification, but if there is only one event "Event1" existing in 24 h , I just want to receive only 1 email in 24h time period.
(my cron schedule:every 10min,predict time period=last 8hours,search time period=last 10 minutes)

Time Event IP
Time1 Event1 IP1

It is like combinations of cronjob, search time period and my predict time period.

Hope it is clear

Thanks
Gokhan

0 Karma

vinitatsky
Communicator

Hi Gokhan,

24 Hours - is our configuration, because we don't want to get another alert for next 24 hours for same IP.
In your case, you have to configure throttling as 8 hours.

Is there any way to ignore alert notification email if search result is as same as the previous result in selected time window=8 hours and every alarm schedule=10 minutes ?
-- Yes. You have to use throttling and configure it to 8 hours.

Time Event IP
Time1 Event1 IP1
Time2 Event2 IP1
Time3 Event3 IP1
-- In above case, it will trigger an alert only if difference between Time1 and Time2 is greater than 8 hours. You will receive 3 alerts (in 24 hours)

Hope this helps.
Vinit

0 Karma

vinitatsky
Communicator

alt text

0 Karma

gyarici
Path Finder

I am not able to see the picture.

0 Karma

vinitatsky
Communicator

Settings from Alerts -

Alert Mode - Once Per Result
Throttling - Checkbox selected for 'After Triggering the alert, don't trigger it again for'
24 Hours
Per Result(s) throttling fields - src_ip

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...