Alerting

Alert based on total for only some rows

jdibble
Explorer

I have a search set up to display HTTP status results with totals and percentage of the total events.

sourcetype="iis" | stats count by sc_status | eventstats sum(count) as total | eval percent = round(100*(count/total),2) . " %" | fields - total

I need to set up an alert for this search based on the results of only some of the columns using a custom condition. I would like to send an alert only if the percentage of the total results is greater than 1% for the combined 404 and 500 errors. I need to know what custom condition search would enable me to do this.

Tags (2)

jdibble
Explorer

Aside from the solution nick posted, I came up with an alternative. (it also includes all 400 and 500 errors, not just 404 and 500) Here's the base search:

sourcetype="iis" | stats count by sc_status | eventstats sum(count) as total | eval percent = round(100*(count/total),2) . " %" | search (sc_status=4* OR sc_status=5*) | eventstats sum(count) as errortotal | eval errper = round(100*(errortotal/total),2) | fields - errortotal, total

With an alert condition of

search errper > 1

This filters it down to just the errors, and the only problem I have with it is that it keeps a column I don't want to display (errper) because removing it from the fields causes the alert not to fire. Still, it does what I need it to do so I can work on finding a way to hide that one column later.

0 Karma

sideview
SplunkTrust
SplunkTrust

I'm sure it's possible with the stats command, and I can update this answer when I have more time, but if it's all the same to you, the custom condition search would be easier if the underlying search was using chart count over foo by sc_status instead of stats count by sc_status.

sourcetype="iis" | eval foo=1 | chart count over foo by sc_status | addtotals | eval percent = round(100*(count/Total),2) . " %" | fields - Total

this trick with chart will put all the data into a single row so it can be much more easily tinkered with later. For instance your custom condition would be just

where 100 * (404+500) > Total

UPDATE::::

If you need it to be a stats search, then I was able to get this working as a custom condition search. I did however need to drop the "%" character from your percent field, to get the percents to add up though. With some time and this as an example I'm sure you can figure out the best way though.

| search sc_status=404 OR sc_status=500 | fields percent | transpose | addtotals | where Total>1

0 Karma

sideview
SplunkTrust
SplunkTrust

OK. I updated my answer. It's possible but clunky.

0 Karma

jdibble
Explorer

Thanks for the answer, but I need to do it with the stats command if it's at all possible.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...