Alerting

How to send alert based on number of occurrences by using email trigger

subramanianp789
New Member

Hi,
I have an requirement that need to schedule the below search query for every 2 mins(it can be given in corn schedule */2 * * * *) but it should not trigger mail immediately even breach threshold and the after specified time limit which will be mentioned in the lookup csv file, consider field as "count_threshold"=3

Now, after 6 mins (ie 2 mins * 3 = 6 mins) the email has to trigger if the search query breach the threshold(or if the results greater than 0)

Below is the existing search:

| eval "Alert Status" = case((' Virtual Bytes'<=manual_threshold3),"NORMAL", (' Virtual Bytes'>manual_threshold4),"CRITICAL", (' Virtual Bytes'>manual_threshold3 AND ' Virtual Bytes'<=manual_threshold4),"WARNING")
| search "Alert Status"="CRITICAL"
| table  Host," PID","Process Name"," Virtual Bytes","Alert Status"

How to achieve this? And how to add the condition to send mail after 6 mins.

Thanks.

0 Karma

woodcock
Esteemed Legend

The way to do this is to write out the results of this search to a lookup file with ... | outputlookup. Then create another scheduled search that runs on a different cron schedule that mines the outputs of the first search using | inputlookup ... and applies your throttling/email logic. This second search (or even a third search) can trim/cleanup the lookup so that it doesn't grow out of control in size.

0 Karma

subramanianp789
New Member

Thanks a lot for your valuable comments. Can you please explain more in detail about the inputlookup. It would be great if you provide me some sample steps to proceed further. Thanks again.

Also, how to pass the value from lookup csv to cron schedule or trigger condition, where to include. Please share your thoughts on this

0 Karma

woodcock
Esteemed Legend

My answer gives all the basic pieces; how you put them together is up to you. The main thing is to put as much of your thresholding and throttling logic inside of the search SPL as possible (try to keep the alert action number of results and is greater than 0).

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...