Deployment Architecture

frozenTimePeriodInSecs points to age of data only in cold bucket or summation of all stages i.e Hot, Warm and Cold ?

damode
Motivator

My client requires 30 day "active" storage and 90 day "cold" storage standard. So, total data should be deleted after 120 days.

below is my current retention setting for main index,
maxHotSpanSecs = 2592000 [hot bucket - 30 days]
maxHotIdleSecs = 0
frozenTimePeriodInSecs = 7776000 [cold bucket - 90 days]
maxTotalDataSizeMB = 250000

Assuming frozenTimePeriodInSecs depends on age of data in cold bucket, data will get rolled from hot bucket after 30 days to warm/cold and after 90 days in cold bucket, it will get rolled to frozen. But I am not sure if my understanding is correct.

Please advise me the correct way to do this.

Tags (1)
0 Karma
1 Solution

harsmarvania57
SplunkTrust
SplunkTrust

Hi,

frozenTimePeriodInSecs applies to warm and cold buckets because Splunk never remove hot bucket directly. So rolling of bucket structure is Hot -> Warm -> Cold.

When you set maxHotSpanSecs=2592000, it doesn’t mean that hot bucket will contains 30 days data because hot-> warm rolling depends on 2 parameter maxHotSpanSecs and maxDataSize whichever hit first. For example you have frozenTimePeriodInSecs set to 7 days but your maxHotSpanSecs is 30 days and your Hot bucket size didn’t reach 750MB(default) then this bucket will not remove after 7 days because it is still hot bucket. Let’s say it converts from hot to warm after 15 days because it hit maxDataSize (default 750 Mb) still it will not remove from Splunk because fromzenTimePeriodInSecs applies to all events in that bucket so until and unless all events in that bucket will be 7 days old this bucket will not remove from splunk.

And warm-> cold bucket rolling depends on maxWarmDBCount.

I hope this helps.

Thanks,
Harshil

View solution in original post

damode
Motivator

Based on answers from this and other posts, I configured some of my indexes in the below way,

maxDataSize = 1000 (~1 day's worth of data)
maxHotSpanSecs = 86401 (to trigger hot --> warm)
maxHotBuckets = 3
maxWarmDBCount = 31 (hot + warm ~ one month's data )
frozenTimePeriodInSecs = 10368000
maxTotalDataSizeMB = 120000 (~120 days)

There is no separate storage for hot and cold. All on same storage.

With the above settings, I can also see buckets rolling to cold, but despite of all of these settings, there is still data in my cold bucket that is older than the frozenTimePeriodInSecs. What can I do to get them into frozen state i.e delete?

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Can you please post folder name from your colddb directory ? And let us know when you applied above configuration?

0 Karma

damode
Motivator

Below are the folders within the colddb directory of in windows index,
db_1512626644_1499845837_0
db_1512629742_1512627226_2
db_1512634103_1512629742_4
I applied this configuration today.

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Based on DB which is currently present in Colddb folder, all 3 DB has latest event from Dec'17. And your frozenTimePeriodInSecs is 120 days means bucket will be removed from splunk when all events present in DB will be older than 120 days OR if you will hit maxTotalDataSizeMB whichever is earlier.

db_1512626644_1499845837_0 -> Earliest event - 12 Jul 2017, Latest Event - 7th Dec 2017 -> Will be removed in Apr'18
db_1512629742_1512627226_2 -> Earliest event - 7 Dec 2017, Latest Event - 7th Dec 2017 -> Will be removed in Apr'18
db_1512634103_1512629742_4 -> Earliest event - 7 Dec 2017, Latest Event - 7th Dec 2017 -> Will be removed in Apr'18

As you have applied setting today now onwards every day your bucket will be roll from hot -> warm and warm -> cold so now onwards your bucket will roll properly but until Apr'18 you will able to see bucket with id 0 from Apr'18 you will able to see only 120 days data.

harsmarvania57
SplunkTrust
SplunkTrust

Hi,

frozenTimePeriodInSecs applies to warm and cold buckets because Splunk never remove hot bucket directly. So rolling of bucket structure is Hot -> Warm -> Cold.

When you set maxHotSpanSecs=2592000, it doesn’t mean that hot bucket will contains 30 days data because hot-> warm rolling depends on 2 parameter maxHotSpanSecs and maxDataSize whichever hit first. For example you have frozenTimePeriodInSecs set to 7 days but your maxHotSpanSecs is 30 days and your Hot bucket size didn’t reach 750MB(default) then this bucket will not remove after 7 days because it is still hot bucket. Let’s say it converts from hot to warm after 15 days because it hit maxDataSize (default 750 Mb) still it will not remove from Splunk because fromzenTimePeriodInSecs applies to all events in that bucket so until and unless all events in that bucket will be 7 days old this bucket will not remove from splunk.

And warm-> cold bucket rolling depends on maxWarmDBCount.

I hope this helps.

Thanks,
Harshil

damode
Motivator

Hi @harsmarvania57, thanks for your input.
If not focussing on sizing attribute of data, just to understand what bucket ageing of data depends on, can you please confirm whether frozenTimePeriodInSecs depends on age of data while it is in cold bucket or its overall age starting from hot-->warm-->cold ?
I checked on splunk-sizing(dot)appspot(dot)com, it seems frozenTimePeriodInSecs means overall age of data in indexer.

Similarly, if we assume age of data in hot bucket hits this number -->30 days considering maxHotSpanSecs=2592000 before it hits maxDataSize, will it roll to warm/cold ?

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

frozenTimePeriodInSecs is depends on the age of overall data in index (Not Indexer), so you can set different frozenTimePeriodInSecs for different indexes in same indexer.

If hot bucket will hit maxHotSpanSecs=2592000 before it hits maxDataSize then yes it will convert it to warm/cold bucket. But maxHotSpanSecs=2592000 doesn't mean that your hot bucket created on 1st Nov will roll on 1st Dec, maxHotSpanSecsapplies to difference between your earliest event and latest event timestamp so in 1 hot bucket earliest event timestamp is 1st Nov 01:00 and on 5th Dec latest event timestamp in same bucket is 28th Nov 23:00 then it will not roll from hot to warm because difference between earliest and latest event timestamp is less than maxHotSpanSecs value.

I hope this clear your query.

damode
Motivator

Hi @Harsh,

Thanks for the explanation.

Is there any drawback to setting maxWarmDBCount=0 if I want to roll bucket straight from hot to cold ?

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi @damode,

There are no drawback to set maxWarmDBCount=0 but there will be no meaning if you are using single disk/mountpath for hot/warm and cold DB, because if all events in single bucket older than frozenTimePeriodInSecs and bucket is in warm directory then splunk will remove that bucket. If you are using different disk/mountpath for hot/warm and cold DB then you need to think about before settingmaxWarmDBCount=0

0 Karma

HiroshiSatoh
Champion

Relationship between Buckets Flow and parameters

Hot → Warm
maxDataSize,maxHotSpanSecs (Default: 750M OR 90Day)

Warm → Cold
maxWarmDBCount (Default: 300)

Hot, Warm, Cold → Frozen
maxTotalDataSizeMB,frozenTimePeriodInSecs (Default: 500G OR 6year)

Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...