All,
What I am trying to do now is get an alert when an index is close to making its retention requirement before running out of space. Anyone have a search they can share?
thank in advance!
-Daniel
Create a lookup called index_retention_goal
with fields indexname
and retentionGoalDays
, and thresholdPct
. Then run this as an alert
:
index=_internal sourcetype=splunkd bucketmover "will attempt to freeze" | rex field=_raw "/splunkdata(?:/[^/]*)?/(?<indexname>[^/]*)/db/db_(?<newestTime>[^_]*)_(?<oldestTime>[^_]*)_.*" | dedup indexname | eval retentionDays=(now()-oldestTime)/(60*60*24) | stats values(retentionDays) as retentionDays by indexname | lookup index_retention_goal indexname | eval currentPct=((100 * retentionDays) / retentionGoalDays) | where currentPct >= thresholdPct
Would this be about the same as wanting a search that compares the two percentages A and B, where
A = the percentage filled an index is, which is consumed space / maximum space
B = the percentage of how much of the expected retention is in the index, which is more complex and should be something like oldest event available / how old the max retention is.
So if you are 50% filled but at only 45% of your retention, you'd want an alert? (Probably doing to run out of space before you hit your retention)
And if you were 50% filled but at 60% of your retention, you're OK? (You have enough space to hold your retention settings, at least probably)
Right?