Splunk Search

How to Set an Alert to Fire on The Total Count?

skoelpin
SplunkTrust
SplunkTrust

I have an alert set which will compare the errors for the current day's previous hour to yesterday's previous hour..

So it will run a search every hour and count the number of errors from 1pm-2pm today and compare them to the number of errors received yesterday from 1pm-2pm. If the number of errors for today's hour is 25% higher than yesterday's hour of errors, then the alert will fire.

I set the search up and it's working as expected, but I believe I misconfigured the alert because I just got 2 alert emails. The first email is showing Calculate Tax errors, then the next alert email is showing the total number of errors. I want the alert to fire if ONLY if the total value for 'TodayLastHour' is 25% greater than 'YesterdayLastHour'.

Tax Call          |   TodayLastHour             | YesterdayLastHour
CalculateTax      |        290                  |      100
LookupTax         |        100                  |       90
TOTAL             |        390                  |      190

Here's my search:

index=vertex7-access RTG_Error="500" earliest=-1h@h latest=@h | stats count AS TodayLastHour by RTG_Tax | addtotals col=t | table RTG_Tax, TodayLastHour | appendcols [search index=vertex7-access RTG_Error="500" earliest=-25h@h latest=-24h@h | stats count AS YesterdayLastHour by RTG_Tax | addtotals col=t | table RTG_Tax, TodayLastHour, YesterdayLastHour] | where TodayLastHour >  1.25 * YesterdayLastHour
1 Solution

woodcock
Esteemed Legend

Go into your alert settings and change When triggered, execute actions to Once and it will give you your expected results.

View solution in original post

woodcock
Esteemed Legend

Go into your alert settings and change When triggered, execute actions to Once and it will give you your expected results.

skoelpin
SplunkTrust
SplunkTrust

Worked as expected, thanks!!

0 Karma

jkat54
SplunkTrust
SplunkTrust

saved search:
index=vertex7-access RTG_Error="500" earliest=-1h@h latest=@h
| stats count AS TodayLastHour
| append [search index=vertex7-access RTG_Error="500" earliest=-25h@h latest=-24h@h | stats count AS YesterdayLastHour]
| where TodayLastHour > 1.25 * YesterdayLastHour

trigger:
if results > 0

run a script: perlscript.that.calls.splunkd.and.triggers.report.email.search

saved report / email search:
index=vertex7-access RTG_Error="500" earliest=-1h@h latest=@h | stats count AS TodayLastHour by RTG_Tax | addtotals col=t | table RTG_Tax, TodayLastHour | appendcols [search index=vertex7-access RTG_Error="500" earliest=-25h@h latest=-24h@h | stats count AS YesterdayLastHour by RTG_Tax | addtotals col=t | table RTG_Tax, TodayLastHour, YesterdayLastHour] | where TodayLastHour > 1.25 * YesterdayLastHour

trigger: always alert

So that the first one triggers a script that runs a saved search which triggers an email. The first one will only run when the where clause is met, and the 2nd search is ran by a perl script via splunkd api call.

somesoni2
SplunkTrust
SplunkTrust

I guess you've configured your alert as per-result hence you get two email. If you just want one 1 alert and that too based on total value, then you would either have to update this current search (you'll loose granularity of TaxCall) OR create a new search. In any case following would be the query to use.

index=vertex7-access RTG_Error="500" earliest=-1h@h latest=@h | stats count AS TodayLastHour by RTG_Tax | addtotals col=t | table RTG_Tax, TodayLastHour | appendcols [search index=vertex7-access RTG_Error="500" earliest=-25h@h latest=-24h@h | stats count AS YesterdayLastHour by RTG_Tax | addtotals col=t | table RTG_Tax, TodayLastHour, YesterdayLastHour] | table TodayLastHour YesterdayLastHour  | stats sum(*) as *  | where TodayLastHour >  1.25 * YesterdayLastHour
0 Karma

richgalloway
SplunkTrust
SplunkTrust

I would create a separate search that only reports the total error count and use that as the basis for the alert.

---
If this reply helps you, Karma would be appreciated.

skoelpin
SplunkTrust
SplunkTrust

I was considering this approach but I'm not sure if this method will return the expected results. I want the alert to show a table (The one I listed above) which will show how many errors in each tax call along with the total errors. But I only want the alert to fire if the total errors from 'TodayLastHour' is 25% more then 'YesterdayLastHour'

0 Karma

jkat54
SplunkTrust
SplunkTrust

I think @richgalloway is correct:

saved search:
index=vertex7-access RTG_Error="500" earliest=-1h@h latest=@h
| stats count AS TodayLastHour
| append [search index=vertex7-access RTG_Error="500" earliest=-25h@h latest=-24h@h | stats count AS YesterdayLastHour]
| where TodayLastHour > 1.25 * YesterdayLastHour

trigger:
if results > 0

run a script: perlscript.that.calls.splunkd.and.triggers.report.email.search.pl

saved report / email search:
index=vertex7-access RTG_Error="500" earliest=-1h@h latest=@h | stats count AS TodayLastHour by RTG_Tax | addtotals col=t | table RTG_Tax, TodayLastHour | appendcols [search index=vertex7-access RTG_Error="500" earliest=-25h@h latest=-24h@h | stats count AS YesterdayLastHour by RTG_Tax | addtotals col=t | table RTG_Tax, TodayLastHour, YesterdayLastHour] | where TodayLastHour > 1.25 * YesterdayLastHour

trigger: always alert

So that the first one triggers a script that runs a saved search which triggers an email. The first one will only run when the where clause is met, and the 2nd search is ran by a perl script via splunkd api call.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...