Splunk Search

Stats / counts in a subsearch for day-over-day delta alerting

cmaier
Explorer

I'm trying to set up an alert based on day-over-day vs. last week for a 5m count. For instance, if certain iis events delta over a certain percentage compared to last week's count.

I can graph the three (Today, Last_Week, and Difference) using this:

index=iis latest=now earliest=-5m sc_status=200 | timechart count as Today | appendcols [search index=iis latest=-10080m earliest=-10085m sc_status=200 | timechart count as Last_Week ] | eval Difference = Today - Last_Week

My question is, how do I turn that into a search I can run every 5 minutes that returns only the "Difference" in percentage form so I can set up an alert if it changes more than an acceptable amount? I've tried using stats, but it seems nesting it in a subsearch creates an issue.

lguinn2
Legend
index=iis latest=now earliest=-24h sc_status=200 | stats count as Today | 
appendcols [search index=iis latest=-168h earliest=-144h sc_status=200 | stats count as Last_Week ] | 
eval Difference = Today - Last_Week | fields + Today Last_Week Difference

When you create the alert, use a custom trigger where Difference > some amount. Set the schedule to run this search every 5 minutes.

You can definitely use stats in a subsearch. In fact, I think that what you want is stats, not timechart. But I am not clear on what you mean by day-over-day vs. last week. Do you want to compare today's count with the same day of last week?

Finally, summary indexing might be helpful. Keep an hour-by-hour summary count, and then your subsearch could run against the summary index VERY quickly.

Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...