Hello,
I would like to get an alert if the indexing volume for an index drops.
I'm thinking something similar this, but I'm not sure how to translate it to Splunk's language:
if (yesterday's total indexing volume for EACH index) is at least 20% less than (the average daily indexing volume for EACH index over -2 to -7 days) then (alert)
Any suggestions?
Starting with the index volume query from the Status menu in Search you can build this:
index=_internal source=*metrics.log group=per_index_thruput | bin span=1d _time | stats sum(kbps) as daily_volume by series _time | eventstats avg(daily_volume) as avg_volume by series | where _time = relative_time(now(), "@d-d") AND daily_volume < avg_volume * 0.8
Run over -7d@d to @d and alert whenever this yields a result.
Starting with the index volume query from the Status menu in Search you can build this:
index=_internal source=*metrics.log group=per_index_thruput | bin span=1d _time | stats sum(kbps) as daily_volume by series _time | eventstats avg(daily_volume) as avg_volume by series | where _time = relative_time(now(), "@d-d") AND daily_volume < avg_volume * 0.8
Run over -7d@d to @d and alert whenever this yields a result.
Yesterday was Sunday, the where is hence looking for a day that was filtered out in the main search.
index=_internal source=*metrics.log group=per_index_thruput (date_wday=monday OR date_wday=tuesday OR date_wday=wednesday OR date_wday=thursday OR date_wday=friday) | bin span=1d _time | stats sum(kbps) as daily_volume by series _time | eventstats avg(daily_volume) as avg_volume by series | where _time = relative_time(now(), "@d-d") AND daily_volume > avg_volume * 0.8
I get no results whether I have a < or >. When removing the (date_wday=monday OR date_wday=tuesday OR date_wday=wednesday OR date_wday=thursday OR date_wday=friday) I do get results with the original search from your first post
Append this to the main search:
(date_wday=monday OR date_wday=tuesday OR date_wday=wednesday OR date_wday=thursday OR date_wday=friday)
Thanks martin!
I'm trying to figure out how to specify weekdays only(as this will trigger every weekend how it is now, because I have an expected large drop in volume during the weekend).
So, I would like to just ignore weekends, and only run the search throughout the week, but the average should only be calculated with weekdays as well.
I see in Splunk Documentation how to specify a specific day of the week, but how could I specify to go 7 days back, but only average the weekdays?