Splunk Search

Stats cannot generate alerts?

dban2005
New Member

I am trying to generate alerts. I have a search query as
index=abc-index host="XYZ123*" collection="AppServer:OrderTracking" counter="Avg. Order Save Time" earliest=-1h
| stats avg(Value) as avgs by host
| where avgs > 5.2

Trying to generate an alert if avgs is larger than 5.2 over the period over last 1 hour. The sample event is like below.

09/01/2017 05:25:19.540 -0700
collection=AppServer:OrderTracking
object=AppServer:OrderTracking
counter="Avg. Order Save Time"
instance=0
Value=8.4719344999999997

I have set up the threshold low at this point so that I can test that the alert is generating. When I am searching with this query, getting at least 3 rows in the Statistic tab with 3 host names and avgs > 5.2
I have set up the alert trigger as
Run on Cron Schedule: */5 * * * *
Number of Hosts: is greater than 0 (I have also tried Number of Results also)
Trigger: Once For each result
Throttle: Checked
Suppress triggering for 15 seconds
Trigger Actions: Alert as well as email to my email address.
With this setting I expected alert emails in every 5 minutes, but not receiving none (BTW other alerts with simple search sending alerts). I am not sure whether I am missing any basics. Any suggestion will be highly appreciated.

0 Karma

woodcock
Esteemed Legend

Take an existing alert that works and clone it. Paste this search's search text into that and change nothing else. Does it work? Probably it will.

0 Karma

dban2005
New Member

Thanks to mmodestino and woodcock to look into this. Yes, I had put */5 * * * * for cron. It showed something like "no event/trigger fired" (though there were a few rows). After spending several hours, i deleted the alert and added as a new from scratch. It worked! Unfortunately, I could not find the error in the previous set up or search. Thanks for your suggestion on throttle.

0 Karma

woodcock
Esteemed Legend

Be sure to click Accept to close the question.

0 Karma

mattymo
Splunk Employee
Splunk Employee

Hi dban2005!

Can I assume you meant */5 * * * * is your cron? So running every 5 mins looking back an hour?

What happens if set it with the action "List in Triggered Alerts"?

I like to use that as the test before relying on email or other means of sending communication of the alert.

Also try setting the time range in the alert config rather than in your search. My hunch is that your scheduled search the alert is using has something set incorrectly.

Also your throttle is unnecessary as it will only fire every 5 mins....(assuming I have ur cron right).

alt text

- MattyMo
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...