Alerting

How do I setup an alert that triggers and take action on second search?

antmob
New Member

I have an alert that triggers when the search returns 0 events for the last couple of hours and sends a slack message. It runs every 5 minutes on cron and looks a few hour back. However, for some reason that I don't know, the alert false triggers some times when it should not and when I manually do the search for which it triggered, I see a bunch of events during that time span. This happens once a month or so.

So if anyone know a solution for this or why this happens that would be great. If not, I'm thinking of changing the alert so that it only triggers if the result is 0 for 2 searches in a row (5 minutes in between) to avoid the false triggers. Is it possible to do this and how?

Labels (3)
0 Karma

Nisha18789
Builder

Hi @antmob 

For investigating the issue, you might want to start with checking the actual search window of the false alert instance using query like :

index=_internal sourcetype=scheduler status=success savedsearch_name=<name of your alert>

And, for 2nd question on 2 consecutive - 0 results, you can join the results of below query with your alert logic or you can use summary index to store the results and join with summary indexed data.

index=_internal sourcetype=scheduler status=success savedsearch_name=<name of your alert> result_count=0

Hope this helps!

 

 

 

 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...