Splunk Search

Splunk indexing volume by index alert

splunkiscool1
Engager

Hello,

I would like to get an alert if the indexing volume for an index drops.

I'm thinking something similar this, but I'm not sure how to translate it to Splunk's language:

if (yesterday's total indexing volume for EACH index) is at least 20% less than (the average daily indexing volume for EACH index over -2 to -7 days) then (alert)

Any suggestions?

0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

Starting with the index volume query from the Status menu in Search you can build this:

index=_internal source=*metrics.log group=per_index_thruput | bin span=1d _time | stats sum(kbps) as daily_volume by series _time | eventstats avg(daily_volume) as avg_volume by series | where _time = relative_time(now(), "@d-d") AND daily_volume < avg_volume * 0.8

Run over -7d@d to @d and alert whenever this yields a result.

View solution in original post

martin_mueller
SplunkTrust
SplunkTrust

Starting with the index volume query from the Status menu in Search you can build this:

index=_internal source=*metrics.log group=per_index_thruput | bin span=1d _time | stats sum(kbps) as daily_volume by series _time | eventstats avg(daily_volume) as avg_volume by series | where _time = relative_time(now(), "@d-d") AND daily_volume < avg_volume * 0.8

Run over -7d@d to @d and alert whenever this yields a result.

martin_mueller
SplunkTrust
SplunkTrust

Yesterday was Sunday, the where is hence looking for a day that was filtered out in the main search.

0 Karma

splunkiscool1
Engager

index=_internal source=*metrics.log group=per_index_thruput (date_wday=monday OR date_wday=tuesday OR date_wday=wednesday OR date_wday=thursday OR date_wday=friday) | bin span=1d _time | stats sum(kbps) as daily_volume by series _time | eventstats avg(daily_volume) as avg_volume by series | where _time = relative_time(now(), "@d-d") AND daily_volume > avg_volume * 0.8

I get no results whether I have a < or >. When removing the (date_wday=monday OR date_wday=tuesday OR date_wday=wednesday OR date_wday=thursday OR date_wday=friday) I do get results with the original search from your first post

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Append this to the main search:

(date_wday=monday OR date_wday=tuesday OR date_wday=wednesday OR date_wday=thursday OR date_wday=friday)
0 Karma

splunkiscool1
Engager

Thanks martin!

I'm trying to figure out how to specify weekdays only(as this will trigger every weekend how it is now, because I have an expected large drop in volume during the weekend).

So, I would like to just ignore weekends, and only run the search throughout the week, but the average should only be calculated with weekdays as well.

I see in Splunk Documentation how to specify a specific day of the week, but how could I specify to go 7 days back, but only average the weekdays?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...