Splunk Search

Stats cannot generate alerts?

dban2005
New Member

I am trying to generate alerts. I have a search query as
index=abc-index host="XYZ123*" collection="AppServer:OrderTracking" counter="Avg. Order Save Time" earliest=-1h
| stats avg(Value) as avgs by host
| where avgs > 5.2

Trying to generate an alert if avgs is larger than 5.2 over the period over last 1 hour. The sample event is like below.

09/01/2017 05:25:19.540 -0700
collection=AppServer:OrderTracking
object=AppServer:OrderTracking
counter="Avg. Order Save Time"
instance=0
Value=8.4719344999999997

I have set up the threshold low at this point so that I can test that the alert is generating. When I am searching with this query, getting at least 3 rows in the Statistic tab with 3 host names and avgs > 5.2
I have set up the alert trigger as
Run on Cron Schedule: */5 * * * *
Number of Hosts: is greater than 0 (I have also tried Number of Results also)
Trigger: Once For each result
Throttle: Checked
Suppress triggering for 15 seconds
Trigger Actions: Alert as well as email to my email address.
With this setting I expected alert emails in every 5 minutes, but not receiving none (BTW other alerts with simple search sending alerts). I am not sure whether I am missing any basics. Any suggestion will be highly appreciated.

0 Karma

woodcock
Esteemed Legend

Take an existing alert that works and clone it. Paste this search's search text into that and change nothing else. Does it work? Probably it will.

0 Karma

dban2005
New Member

Thanks to mmodestino and woodcock to look into this. Yes, I had put */5 * * * * for cron. It showed something like "no event/trigger fired" (though there were a few rows). After spending several hours, i deleted the alert and added as a new from scratch. It worked! Unfortunately, I could not find the error in the previous set up or search. Thanks for your suggestion on throttle.

0 Karma

woodcock
Esteemed Legend

Be sure to click Accept to close the question.

0 Karma

mattymo
Splunk Employee
Splunk Employee

Hi dban2005!

Can I assume you meant */5 * * * * is your cron? So running every 5 mins looking back an hour?

What happens if set it with the action "List in Triggered Alerts"?

I like to use that as the test before relying on email or other means of sending communication of the alert.

Also try setting the time range in the alert config rather than in your search. My hunch is that your scheduled search the alert is using has something set incorrectly.

Also your throttle is unnecessary as it will only fire every 5 mins....(assuming I have ur cron right).

alt text

- MattyMo
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...