Getting Data In

Remove data when it ages

shangshin
Builder

Hi, I saw the doc on how to remove data when it ages.
Most of my log data goes to the default index db which is "main"

So assuming I want to index data for ONLY the latest one hour, I added these 2 lines in /etc/system/default/indexes.conf

[main]
frozenTimePeriodInSecs = 3600

On top of that I also created a new file /etc/system/local/indexes.conf with the same 2 lines.

[main]
frozenTimePeriodInSecs = 3600

Then I restarted splunk server but I still saw data older than 1 hour ago.

Can anyone shed some light on it? Thank you!!!

0 Karma
1 Solution

yannK
Splunk Employee
Splunk Employee

Hi Shangshin

Keeping only one hour of data is pretty short, what is your use case ?

A short frozenTimePeriodInSecs will not be enough to ensure that the data is frozen quickly.
This is because the events are written in buckets, a bucket have a certain span of time, or a maximum size. The bucket being written are in hot state, and roll to warm (or cold) state when they became read only. But only warm or cold state buckets can actually be frozen, and only if all the events their contains are older than the frozenTimePeriodInSecs or the maximum size of the index is reached. Then the ones containing the oldest events will be frozen.

The maxHotSpanSecs parameter can be used to limit the span times for your buckets, and force them to roll from hot to warm more often. Then the frozenTimePeriodInSecs will kick in.

see http://wiki.splunk.com/Deploy:BucketRotationAndRetention
http://docs.splunk.com/Documentation/Splunk/5.0/Admin/Indexesconf

View solution in original post

yannK
Splunk Employee
Splunk Employee

Hi Shangshin

Keeping only one hour of data is pretty short, what is your use case ?

A short frozenTimePeriodInSecs will not be enough to ensure that the data is frozen quickly.
This is because the events are written in buckets, a bucket have a certain span of time, or a maximum size. The bucket being written are in hot state, and roll to warm (or cold) state when they became read only. But only warm or cold state buckets can actually be frozen, and only if all the events their contains are older than the frozenTimePeriodInSecs or the maximum size of the index is reached. Then the ones containing the oldest events will be frozen.

The maxHotSpanSecs parameter can be used to limit the span times for your buckets, and force them to roll from hot to warm more often. Then the frozenTimePeriodInSecs will kick in.

see http://wiki.splunk.com/Deploy:BucketRotationAndRetention
http://docs.splunk.com/Documentation/Splunk/5.0/Admin/Indexesconf

shangshin
Builder

Thanks for detail explanation. It's very helpful.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...