Alerting

How to create an alert to trigger when an index is close to meeting its retention requirement before running out of space?

daniel333
Builder

All,

What I am trying to do now is get an alert when an index is close to making its retention requirement before running out of space. Anyone have a search they can share?

thank in advance!
-Daniel

0 Karma

woodcock
Esteemed Legend

Create a lookup called index_retention_goal with fields indexname and retentionGoalDays, and thresholdPct. Then run this as an alert:

index=_internal sourcetype=splunkd bucketmover "will attempt to freeze" | rex field=_raw "/splunkdata(?:/[^/]*)?/(?<indexname>[^/]*)/db/db_(?<newestTime>[^_]*)_(?<oldestTime>[^_]*)_.*" | dedup indexname | eval retentionDays=(now()-oldestTime)/(60*60*24) | stats values(retentionDays) as retentionDays by indexname | lookup index_retention_goal indexname | eval currentPct=((100 * retentionDays) / retentionGoalDays) | where currentPct >= thresholdPct
0 Karma

Richfez
SplunkTrust
SplunkTrust

Would this be about the same as wanting a search that compares the two percentages A and B, where

A = the percentage filled an index is, which is consumed space / maximum space
B = the percentage of how much of the expected retention is in the index, which is more complex and should be something like oldest event available / how old the max retention is.

So if you are 50% filled but at only 45% of your retention, you'd want an alert? (Probably doing to run out of space before you hit your retention)

And if you were 50% filled but at 60% of your retention, you're OK? (You have enough space to hold your retention settings, at least probably)

Right?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...