Deployment Architecture

limit number of days of index data kept

john_howley
Path Finder

Can I automatically set a limit that keeps only say the last month of indexed data for example with nmon data. Or do I just delete the directories directly in the filesystem?

Tags (2)

john_howley
Path Finder

Hi yannk, thanks for the answer - very useful and I think I have a handle on it now in terms of the various states. I just have one more question - if we were to set maxTotalDataSizeMB and frozenTimePeriodInSecs to ensure that we don't keep too much data in active indexes, but due to the default settings within indexes.conf meant that data had not moved from WARM to COLD would that data be moved from WARM to FROZEN directly?

0 Karma

yannK
Splunk Employee
Splunk Employee

the lifetime of a bucket is :

  1. hot (read and write) - on the homePath
  2. then warm or cold - on the homePath or coldPath respectively
  3. then frozen (deleted or archived is a coldtoFrozenDir or script is defined)
  4. and eventually thawed if restored from frozen - in the thawedPath

So a bucket can go directly from warm to frozen without being cold.
The Cold state is optional, you may end up with an empty coldPath with the splunk defaults. And is only useful if you are using homePath and coldPath on different partitions for space reasons.

0 Karma

yannK
Splunk Employee
Splunk Employee

Read indexes.conf specifications. You will need to edit a config file.
see http://docs.splunk.com/Documentation/Splunk/6.2.3/Indexer/Setaretirementandarchivingpolicy

setup a ** frozenTimePeriodInSecs** to be the day limit. (default 6 years)

and if you retention is very low (less than months) , you may also want to reduce the maxHotSpanSecs (maximum hot bucket timespan default to 90 days) to force them to rotate more often. (as hot buckets cannot be frozen)

Remark : do not use exactly 1 hour or 1 day for maxHotSpanSecs, It's best to use one week, it prevent too many buckets to be created if your events are not received chronologically.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...