I have an alert set which will compare the errors for the current day's previous hour to yesterday's previous hour..
So it will run a search every hour and count the number of errors from 1pm-2pm today and compare them to the number of errors received yesterday from 1pm-2pm. If the number of errors for today's hour is 25% higher than yesterday's hour of errors, then the alert will fire.
I set the search up and it's working as expected, but I believe I misconfigured the alert because I just got 2 alert emails. The first email is showing Calculate Tax errors, then the next alert email is showing the total number of errors. I want the alert to fire if ONLY if the total value for 'TodayLastHour' is 25% greater than 'YesterdayLastHour'.
Tax Call | TodayLastHour | YesterdayLastHour
CalculateTax | 290 | 100
LookupTax | 100 | 90
TOTAL | 390 | 190
Here's my search:
index=vertex7-access RTG_Error="500" earliest=-1h@h latest=@h | stats count AS TodayLastHour by RTG_Tax | addtotals col=t | table RTG_Tax, TodayLastHour | appendcols [search index=vertex7-access RTG_Error="500" earliest=-25h@h latest=-24h@h | stats count AS YesterdayLastHour by RTG_Tax | addtotals col=t | table RTG_Tax, TodayLastHour, YesterdayLastHour] | where TodayLastHour > 1.25 * YesterdayLastHour
Go into your alert settings and change When triggered, execute actions
to Once
and it will give you your expected results.
Go into your alert settings and change When triggered, execute actions
to Once
and it will give you your expected results.
Worked as expected, thanks!!
saved search:
index=vertex7-access RTG_Error="500" earliest=-1h@h latest=@h
| stats count AS TodayLastHour
| append [search index=vertex7-access RTG_Error="500" earliest=-25h@h latest=-24h@h | stats count AS YesterdayLastHour]
| where TodayLastHour > 1.25 * YesterdayLastHour
trigger:
if results > 0
run a script: perlscript.that.calls.splunkd.and.triggers.report.email.search
saved report / email search:
index=vertex7-access RTG_Error="500" earliest=-1h@h latest=@h | stats count AS TodayLastHour by RTG_Tax | addtotals col=t | table RTG_Tax, TodayLastHour | appendcols [search index=vertex7-access RTG_Error="500" earliest=-25h@h latest=-24h@h | stats count AS YesterdayLastHour by RTG_Tax | addtotals col=t | table RTG_Tax, TodayLastHour, YesterdayLastHour] | where TodayLastHour > 1.25 * YesterdayLastHour
trigger: always alert
So that the first one triggers a script that runs a saved search which triggers an email. The first one will only run when the where clause is met, and the 2nd search is ran by a perl script via splunkd api call.
I guess you've configured your alert as per-result hence you get two email. If you just want one 1 alert and that too based on total value, then you would either have to update this current search (you'll loose granularity of TaxCall) OR create a new search. In any case following would be the query to use.
index=vertex7-access RTG_Error="500" earliest=-1h@h latest=@h | stats count AS TodayLastHour by RTG_Tax | addtotals col=t | table RTG_Tax, TodayLastHour | appendcols [search index=vertex7-access RTG_Error="500" earliest=-25h@h latest=-24h@h | stats count AS YesterdayLastHour by RTG_Tax | addtotals col=t | table RTG_Tax, TodayLastHour, YesterdayLastHour] | table TodayLastHour YesterdayLastHour | stats sum(*) as * | where TodayLastHour > 1.25 * YesterdayLastHour
I would create a separate search that only reports the total error count and use that as the basis for the alert.
I was considering this approach but I'm not sure if this method will return the expected results. I want the alert to show a table (The one I listed above) which will show how many errors in each tax call along with the total errors. But I only want the alert to fire if the total errors from 'TodayLastHour' is 25% more then 'YesterdayLastHour'
I think @richgalloway is correct:
saved search:
index=vertex7-access RTG_Error="500" earliest=-1h@h latest=@h
| stats count AS TodayLastHour
| append [search index=vertex7-access RTG_Error="500" earliest=-25h@h latest=-24h@h | stats count AS YesterdayLastHour]
| where TodayLastHour > 1.25 * YesterdayLastHour
trigger:
if results > 0
run a script: perlscript.that.calls.splunkd.and.triggers.report.email.search.pl
saved report / email search:
index=vertex7-access RTG_Error="500" earliest=-1h@h latest=@h | stats count AS TodayLastHour by RTG_Tax | addtotals col=t | table RTG_Tax, TodayLastHour | appendcols [search index=vertex7-access RTG_Error="500" earliest=-25h@h latest=-24h@h | stats count AS YesterdayLastHour by RTG_Tax | addtotals col=t | table RTG_Tax, TodayLastHour, YesterdayLastHour] | where TodayLastHour > 1.25 * YesterdayLastHour
trigger: always alert
So that the first one triggers a script that runs a saved search which triggers an email. The first one will only run when the where clause is met, and the 2nd search is ran by a perl script via splunkd api call.