Alerting

Email alert not triggering

twiggle
Explorer

Hi I need help with my email alerts.

I basically need to have an email alerting me that one of my process which I am logging is taking more than 2 hours or x hours.

So I have the basic query set up and let's say it is QUERY.

I've made the following alert from the following query:

QUERY | eval result=if(x>2,"YES","NO") | table result
where x is the current time since the process started (in hours).

I then saved this query as an alert and used the following settings:
Alert type: real time
Trigger condition: custom
Custom condition: search result=YES
in: 2 day(s)

I verified that the search query:

QUERY | eval result=if(x>2,"YES","NO") | table result | search result=YES

gives me a result if the time taken is more than 2 hours however it doesn't trigger an email alert.

Anyone can give me an idea of what I did wrong or where I can go from here?

Tags (2)
0 Karma
1 Solution

woodcock
Esteemed Legend

First of all, unless you deliberately engineered your Splunk cluster for RealTime searches, DO NOT USE THEM; you will destroy performance on your entire cluster. I would run your search instead every 5 or 10 minutes over the last X hours. Next, I would use "QUERY | where x>2" and then use trigger conditions for "number of results > 0".

View solution in original post

woodcock
Esteemed Legend

First of all, unless you deliberately engineered your Splunk cluster for RealTime searches, DO NOT USE THEM; you will destroy performance on your entire cluster. I would run your search instead every 5 or 10 minutes over the last X hours. Next, I would use "QUERY | where x>2" and then use trigger conditions for "number of results > 0".

twiggle
Explorer

How do you get it to search every 5 or 10 minutes?

I looked at the schedule alert type and under the 'Time Range' there's only 'Run every hour', '.. Day', '... Week' etc.

0 Karma

twiggle
Explorer

Ah ok, using the cron notation for scheduled alerts right?

*/5 * * * * or */10 * * * *

0 Karma

woodcock
Esteemed Legend

Yes, "/5" works.

0 Karma

jeremiahc4
Builder

Are you verifying the search in real-time the same way you are scheduling it? I wonder if real-time can't keep track that far out.

0 Karma

twiggle
Explorer

Yes that's what I did to verify it. I did it that way as I read that the custom condition applies the query that you insert above the base query.

Which in this case is: QUERY | eval result=if(x>2,"YES","NO") | table result

I did ensure that the real-time search looks at records beyond 2 hours.

I'll look into what @woodcock mentioned. That seems to be a better alternative.

0 Karma

MichaelPriest
Communicator

Try with == instead of =, I'm not sure if this will help?

0 Karma

twiggle
Explorer

Nope, that didn't do the trick.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...