Getting Data In

how to configure retirement policy?

giovere
Path Finder

I'm trying to make indexes retire after 60 seconds, here is how my indexes.conf looks like:

[default]

frozenTimePeriodInSecs = 60

rotatePeriodInSecs = 10

If I restart Splunk, it indeed removes all events older than 60 seconds, but I could not achieve the same without restarting Spunk. Am I missing any additional rotate parameter?

If the problem is related to maximum buket sizes (defualt 300) is there a way to make them age dependent?

Tags (1)
0 Karma
1 Solution

yannK
Splunk Employee
Splunk Employee

Why would you do such a thing ?
what is the purpose of keeping 60 seconds of data in splunk ?
Also You shouldn't setup default rules, they will apply to ALL your indexes, including the internal splunk ones, and all your summary indexes.

the reasons why your rule doesn't remove all the old events instantly are :

  • the events are stored in buckets per time period, and a bucket is been rotated only if all the events of a buckets are older than the retention period.
  • also a hot bucket is not immediately frozen, it has to be at least rotated to warm.

The reason why it worked after a restart :

  • When you restart splunk (since 4.2), all the hot buckets are rotated, therefore your settings apply to them.

The only clean way to achieve your goal is to change the maximum size and span of a bucket to keep them small.
see maxDataSize and maxHotSpanSecs.

see explanations
http://www.splunk.com/wiki/Deploy:BucketRotationAndRetention

View solution in original post

yannK
Splunk Employee
Splunk Employee

Why would you do such a thing ?
what is the purpose of keeping 60 seconds of data in splunk ?
Also You shouldn't setup default rules, they will apply to ALL your indexes, including the internal splunk ones, and all your summary indexes.

the reasons why your rule doesn't remove all the old events instantly are :

  • the events are stored in buckets per time period, and a bucket is been rotated only if all the events of a buckets are older than the retention period.
  • also a hot bucket is not immediately frozen, it has to be at least rotated to warm.

The reason why it worked after a restart :

  • When you restart splunk (since 4.2), all the hot buckets are rotated, therefore your settings apply to them.

The only clean way to achieve your goal is to change the maximum size and span of a bucket to keep them small.
see maxDataSize and maxHotSpanSecs.

see explanations
http://www.splunk.com/wiki/Deploy:BucketRotationAndRetention

giovere
Path Finder

maxHotIdleSecs did the trick, so finally it looks like this:

[main]

frozenTimePeriodInSecs = 10

rotatePeriodInSecs = 10

maxHotIdleSecs = 120

indexes spend 120 secs in hot buckets (starting from the last write action to a bucket), then rolled to warm spend 10 seconds there and finally they are deleted. If you sense any pitfall here, please let me know I'm quite new to all Splunk things.

0 Karma

giovere
Path Finder

thanks for comments, of course 60 secs is not real life problem, I just wanted test it on lower time limit to make sure it is working properly.

In other words my goal is: make indexes age independently of their size or position in a bucket, avoiding rotation between hot,warm,cold (just delete when it is old).

My current version looks like this:

[main]

frozenTimePeriodInSecs = 60

rotatePeriodInSecs = 10

maxHotSpanSecs = 10

Current index contains only date > date.log which is old enough to be removed, but is not. Any suggestions will help. Thanks ...

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

I think that you're perhaps fundamentally misunderstanding the the retirement/freezing settings. I don't think that you should even attempt to achieve the goal of deleting data once it is more than 60 seconds old, and I doubt that you will be able to guarantee that no matter what you set maxDataSize or maxHotSpanSecs to.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...