Can "maxTotalDataSizeMB" & "frozenTimePeriodInSecs" be combined for Index config ?

[pkumar9610]

HI Friends,

I am using below config for creating Indexes in both my QA & Production Cluster. At this point, I am only using retention period for Indexes but it is not helping in capacity management. Can I add frozenTimePeriodInSecs to this config so that, if it reaches capacity limit, it will take care of it ?

[ship]
homePath   = volume:primary/ship/db
coldPath   = volume:primary/ship/colddb
thawedPath = $SPLUNK_DB/ship/thaweddb
frozenTimePeriodInSecs=10368000

frozenTimePeriodInSecs when it reaches the capacity limit, does it remove the old logs and continue Indexing new logs just like frozenTimePeriodInSecs? Or does it just stop Indexing when it reaches the limit ?

Thanks,
-Prashanth

[Rob2520 · Accepted solution]

"maxTotalDataSizeMB" takes precedence over other "frozenTimePeriodInSecs".

If the index grows beyond maxTotalDataSizeMB megabytes before frozenTimePeriodInSecs seconds have passed, data could prematurely roll to frozen, and if frozenTimePeriodInSecs comes first, then data will be rolled to frozen as well.

To answer your question: YES. If your raw data reaches "frozenTimePeriodInSecs" seconds, then you will start loosing old data and continue indexing new data.

[sakthiganesht]

What happens when the frozenTimePeriodInSecs is reached but maxTotalDataSizeMB is not reached? Will it freeze indexed data older than frozenTimePeriodInSecs or continue to store them in colddb till the size reaches maxTotalDataSizeMB ?

[ddrillic]

maxTotalDataSizeMB and frozenTimePeriodInSecs coexist ; -)

[pkumar9610]

I have updated my config to have both frozenTimePeriodInSecs & maxTotalDataSizeMB, but I don't see it is limiting to 1024MB. IS some thing wrong with my config here ?

[sse-router-qa]
homePath = volume:primary/sse-router-qa/db
coldPath = volume:primary/sse-router-qa/colddb
thawedPath = $SPLUNK_DB/sse-router-qa/thaweddb
frozenTimePeriodInSecs=172800
maxTotalDataSizeMB = 1024
maxHotBuckets = 6

[pkumar9610]

Thank you for the info.

Lets say for example if I have set maxTotalDataSizeMB=100GB, is this 100GB is the RAW data size or after Splunk does it compression.

IF it is RAW data, how much size will it be after compression ?
And do I need to do this capacity planning with the RAW data size coming ?

[jtacy]

maxTotalDataSizeMB is the maximum total size of all buckets associated with an index. This includes the indexes (tsidx files) and compressed raw data (journal.gz). It also includes the buckets replicated from other indexers in your cluster.

The compression ratio for raw data varies, but you'll probably find that the index portion of the bucket is generally larger than the compressed raw data portion. An index can vary dramatically in size relative to the raw data depending on the number of unique terms (segments) that Splunk needs to index. That's especially true if you're using any index-time field extractions. Splunk buckets are just collections of files so you can look at them to compare the ratio for your own data. Splunk's official documentation about this is at:
https://docs.splunk.com/Documentation/Splunk/7.1.3/Capacity/Estimateyourstoragerequirements