Is this 180 day retention policy configuration in indexes.conf appropriate?

[sidekix24]

Hi,

We are are setting up our indexes to all have a retention policy of 180 total days. 10 days in hot/warm and 170 in cold.

Below is a sample stanza that we plan to setup for each index. Can we get some suggestions if this will fit our needs or if there's a better approach?

[ai_gomez_1]
repFactor=auto
homePath=/opt/splunkdb/ai_gomez_1/db
coldPath=/mnt2/splunkdb-cold/ai_gomez_1/colddb
thawedPath=/mnt2/splunkdb-cold/ai_gomez_1/thaweddb
# Roll hot buckets to warm (after 7-days)
maxHotSpanSecs=604800
# Roll warm buckets to cold when warm bucket count reaches this value (default=300)
maxWarmDBCount=10
# (Delete data) roll warm/cold buckets to frozen bucket (after 6-months)
frozenTimePeriodInSecs=15552000
# Max size of index in MB, max value: 4294967295
maxTotalDataSizeMB=4294967295

Thanks

[s2_splunk · Accepted solution]

The transition from hot to warm is simply a transition from read/write buckets to read-only buckets. The transition from WARM to COLD happens based on either reaching the maxWarmDBCount or homePath.maxDataSizeMB. So, you need to use your daily ingest volume to calculate what those settings should be to get about 10 days worth of data kept in HOT/WARM. If you are using auto_high_volume for maxDataSize, your buckets will be ~10GB each. The rest is math. 😉

The transition from COLD to FROZEN happens based on either size constraints or time (frozenTimePeriodInSecs). Note that this is not exact as data ages out by bucket, so the youngest event in any given bucket needs to be older than frozenTimePeriodInSecs to cause the bucket to be rolled to FROZEN.

I would recommend you use the Splunk sizing tool to dial in your settings; it will give you proper indexes.conf settings at the very bottom.

[sidekix24]

Thanks ssievert,

We did use that tool as well and it came up this:

[main]
homePath = volume:hotwarm_cold/defaultdb/db
coldPath = volume:hotwarm_cold/defaultdb/colddb
thawedPath = $SPLUNK_DB/defaultdb/thaweddb
homePath.maxDataSizeMB = 409600
coldPath.maxDataSizeMB = 7372800
maxWarmDBCount = 4294967295
frozenTimePeriodInSecs = 16416000
maxDataSize = auto_high_volume

My question is....it says that this is the setting they came up with assuming all data is going to the main index. We will be using multiple indexes so would we use these same numbers for each index or do we have to break down the numbers using the daily ingestion of each index?

Thanks

[s2_splunk]

Since indexes.conf settings apply on a per-index basis, you would want to calculate per daily ingest for all data targeted at a given index.

[sidekix24]

ok...that's what I figured.

Thanks

[s2_splunk]

NP. Please accept my answer if it resolves your question. Thanks!