Ideal indexes.conf

[nawazns5038]

Hi,

Can I please know the ideal configurations for indexes.conf ?

Should we include parameters like homePath.maxDataSizeMB , coldPath.maxDataSizeMB etc. ?

Or is it enough to specify only frozenTimePeriodInSecs

[ddrillic]

The two important ones are frozenTimePeriodInSecs and maxTotalDataSizeMB with the default of 1/2 TB.

[deepashri_123]

Hey nawazns5038,

Your total retention period includes hot warm and cold buckets. There are default parameters in splunk from rolling hot to warm buckets and warm to cold, although can be changed. However frozenTimePeriodInSecs parameter decides when the bucket has to be deleted from Splunk, so adding this parameter should do the trick.

You can refer the doc below:
http://wiki.splunk.com/Deploy:BucketRotationAndRetention

Let me know if this helps!!

[nawazns5038]

Hi Deepa

homePath.maxDataSizeMB = 2000000
coldPath.maxDataSizeMB = 15664000
frozenTimePeriodInSecs = 2505600
maxTotalDataSizeMB = 34664000
maxDataSize = auto_high_volume
repFactor = auto

How does Splunk roll the buckets from hot to warm warm to cold based on the settings I mentioned above.

[nawazns5038]

If I just mention frozenTimePeriodInSecs as 30 days , how does Splunk roll the buckets exactly so that the data gets deleted in 30 days ?

Will adding the parameters like homePath.maxDataSizeMB , coldPath.maxDataSizeMB effect the rolling of buckets ?

[deepashri_123]

You can refer this link to understand how bucket ages:
http://docs.splunk.com/Documentation/Splunk/7.0.2/Indexer/HowSplunkstoresindexes

When you mention frozenTimePeriodInSecs, this parameter decides the retention of bucket from coldpath and deletes the file from splunk

[nawazns5038]

It does not mention the way how Splunk rolls the buckets if we just mention the frozenTimePeriodInSecs .

The question is pretty straight forward, we set frozenTimePeriodInSecs = 30 days .... how does splunk roll the buckets so that data gets deleted by 30 days from the day it comes in .

you have mentioned that the "When you mention frozenTimePeriodInSecs, this parameter decides the retention of bucket from coldpath and deletes the file from splunk "
but there are stages in between which the data goes through to reach cold state.

[nawazns5038]

What if the data volume is low and is present only in the hot and warm buckets itself and didn't come into cold still and the period has exceeded 30 days ?
So the retention policy will not apply in this case ??

[damiensurat]

This is a hard question ion to answer as the parameters you mention should be set to accommodate data size and retention time of your data. Being that data in Splunk index buckets will start to roll from hot to warm to cold and then frozen based on both the size and /or retention time I would suggest that you understand how much data you need to capture and retain as well as how long the data should remain available in the index. You can always increase / decrease the size and retention time as needed.