How do I get 1 alert per condition rather than getting several?

[skoelpin]

I have a search which uses the transaction command to group 2 events together. I then added a where clause to sort the events that had a duration>2 seconds.. I then saved the search as an alert.. Now every time the duration>2 seconds, I get 3 identical emails with a different trigger time for each. I'm viewing the logs in real time and verified that only 1 event came in that was greater than 2 seconds, even though I'm getting 3 emails. How can I change this where I will only get 1 email anytime duration>2?

My Search

index=unleashed Call="<CreateOrder*"  | transaction GUID startswith="fterReceiveRequest" endswith="BeforeSendReply" | where duration>2

[aljohnson_splun · Accepted solution]

Hmm. It makes sense to not want to throttle.

I'm formulating the problem to myself as how do we represent event uniqueness - you want to search in the last -15m for example but run this search every 5 minutes? I still think it makes more sense just to run the search every 5 minutes and search over the last 5 minutes. That way, there aren't any duplicates that are found. Because right now, it sounds like you are running a search that looks back farther than the cron schedule - and I think that is the main problem.

I'm sure there are better ways to do this - here is the first thing that came to mind:

 index=unleashed Call="<CreateOrder*"  
| transaction GUID startswith="fterReceiveRequest" endswith="BeforeSendReply" 
| where duration > 2
| eval hash = md5(GUID + _time)
| eval time = _time
| search NOT [| inputlookup hashed.csv | return hash]
| where _time > now() - (15 * 60) # NOTE : this could be whatever your time range is
| outputlookup hashed.csv

[skoelpin]

I took a different approach and set the search to look within a 1 minute time frame then set the cron schedule for 1 minute and this works much better. You gave me the idea from your original post so I'll give you credit for the answer. Thanks for your help!

[aljohnson_splun]

Awesome. That seems like the right approach - same cron & timeframe. Glad you figured it out.

[somesoni2]

Is it a real-time saved search OR you're executing at a certain interval?

[skoelpin]

Can I get an update on this?

[aljohnson_splun]

If its every 5 minutes:

00:00 - 1 event matched - triggered
00:05 - 1 event matched - triggered

00:10 - 1 event matched - triggered
00:15 - 0 events matched - nothing triggered

all three are the same event?

It sounds like you have the window of the search too large - so that multiple runs pick up the same information. You can change the window of your root search, or you could use alert throttling to make it not trigger for 10 minutes after (less desirable solution IMO).

[skoelpin]

Thanks for the quick response! Would there be anyway to keep the 5 minute window but only have it alert on new events which fit the criteria rather than alerting on events that fall into that 5 minute window?

I'd rather not throttle because it means we could miss events where the duration is greater than 2 seconds. We primarily want this for getting a lot of alerts in a short time period which would let us know there's a problem.

If there's no way to have it alert on only new events in that 5 minute window, then would you recommend me decreasing that 5 window to a 30 second window?

[skoelpin]

It's on a cron schedule for every 5 minutes. We sometimes have multiple alerts within that 5 minute time window, and sometimes we can have 0 or 1 alert within that 5 minute time window