Getting Data In

Remove data when it ages

shangshin
Builder

Hi, I saw the doc on how to remove data when it ages.
Most of my log data goes to the default index db which is "main"

So assuming I want to index data for ONLY the latest one hour, I added these 2 lines in /etc/system/default/indexes.conf

[main]
frozenTimePeriodInSecs = 3600

On top of that I also created a new file /etc/system/local/indexes.conf with the same 2 lines.

[main]
frozenTimePeriodInSecs = 3600

Then I restarted splunk server but I still saw data older than 1 hour ago.

Can anyone shed some light on it? Thank you!!!

0 Karma
1 Solution

yannK
Splunk Employee
Splunk Employee

Hi Shangshin

Keeping only one hour of data is pretty short, what is your use case ?

A short frozenTimePeriodInSecs will not be enough to ensure that the data is frozen quickly.
This is because the events are written in buckets, a bucket have a certain span of time, or a maximum size. The bucket being written are in hot state, and roll to warm (or cold) state when they became read only. But only warm or cold state buckets can actually be frozen, and only if all the events their contains are older than the frozenTimePeriodInSecs or the maximum size of the index is reached. Then the ones containing the oldest events will be frozen.

The maxHotSpanSecs parameter can be used to limit the span times for your buckets, and force them to roll from hot to warm more often. Then the frozenTimePeriodInSecs will kick in.

see http://wiki.splunk.com/Deploy:BucketRotationAndRetention
http://docs.splunk.com/Documentation/Splunk/5.0/Admin/Indexesconf

View solution in original post

yannK
Splunk Employee
Splunk Employee

Hi Shangshin

Keeping only one hour of data is pretty short, what is your use case ?

A short frozenTimePeriodInSecs will not be enough to ensure that the data is frozen quickly.
This is because the events are written in buckets, a bucket have a certain span of time, or a maximum size. The bucket being written are in hot state, and roll to warm (or cold) state when they became read only. But only warm or cold state buckets can actually be frozen, and only if all the events their contains are older than the frozenTimePeriodInSecs or the maximum size of the index is reached. Then the ones containing the oldest events will be frozen.

The maxHotSpanSecs parameter can be used to limit the span times for your buckets, and force them to roll from hot to warm more often. Then the frozenTimePeriodInSecs will kick in.

see http://wiki.splunk.com/Deploy:BucketRotationAndRetention
http://docs.splunk.com/Documentation/Splunk/5.0/Admin/Indexesconf

shangshin
Builder

Thanks for detail explanation. It's very helpful.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...