Splunk Search

Stats / counts in a subsearch for day-over-day delta alerting

cmaier
Explorer

I'm trying to set up an alert based on day-over-day vs. last week for a 5m count. For instance, if certain iis events delta over a certain percentage compared to last week's count.

I can graph the three (Today, Last_Week, and Difference) using this:

index=iis latest=now earliest=-5m sc_status=200 | timechart count as Today | appendcols [search index=iis latest=-10080m earliest=-10085m sc_status=200 | timechart count as Last_Week ] | eval Difference = Today - Last_Week

My question is, how do I turn that into a search I can run every 5 minutes that returns only the "Difference" in percentage form so I can set up an alert if it changes more than an acceptable amount? I've tried using stats, but it seems nesting it in a subsearch creates an issue.

lguinn2
Legend
index=iis latest=now earliest=-24h sc_status=200 | stats count as Today | 
appendcols [search index=iis latest=-168h earliest=-144h sc_status=200 | stats count as Last_Week ] | 
eval Difference = Today - Last_Week | fields + Today Last_Week Difference

When you create the alert, use a custom trigger where Difference > some amount. Set the schedule to run this search every 5 minutes.

You can definitely use stats in a subsearch. In fact, I think that what you want is stats, not timechart. But I am not clear on what you mean by day-over-day vs. last week. Do you want to compare today's count with the same day of last week?

Finally, summary indexing might be helpful. Keep an hour-by-hour summary count, and then your subsearch could run against the summary index VERY quickly.

Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...