Getting Data In

How to force retention time pruning

hartfoml
Motivator

I have my frozen time set like this frozenTimePeriodInSecs = 47304000 (1.5 years)
yet when I do this search

| metadata index=foo type=hosts | stats max(lastTime) as lastTime, min(firstTime) as firstTime | convert ctime(*Time)

my "firstTime" is more than two years from my "lastTime"

lastTime               firstTime
02/06/2017 07:10:40     01/16/2015 09:18:53

this is more than 2 years of data.
How can I force retention time pruning or find out why pruning it not running correctly?

0 Karma

ddrillic
Ultra Champion

So, it's interesting to find out what the value of maxHotSpanSecs is now and what it was 1.5 years ago, because its default value of 1 day in seconds, enforces rotation of the bucket on a daily basis. Keep in mind that this value for a slow growing index can produce lots of small buckets...

0 Karma

twinspop
Influencer

I'm not sure about that default. On all my 6.5.1 systems, the default is 7776000, which is 90 days.

0 Karma

ddrillic
Ultra Champion

Really interesting, because I'm pretty sure on older versions it was a day...

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi hartfoml,
events deletion is related to buckets time period: a bucket is deleted when the latest bucket's event is out of the retention period, it isn't possible to force the bucket's deletion.

Bye.
Giuseppe

0 Karma

hartfoml
Motivator

Giuseppe, @cusello Thanks for the response. I have the retention set to 1.5 years and there is more than 2 years of data. This is a high volume index "Firewall" logs. Since I have 2 yeas worth of data, bucket data would have to span more than 6 moths in order not to be deleted in the 1.5 year time frame. this seems unlikely to me.

I wish there was a way of looking into the buckets to see which ones should be removed and are not being removed.

Thanks for your response.

0 Karma

ddrillic
Ultra Champion

You see, each bucket has two epoch time stamps, in their file name, which define their time interval. So, you can check them on the file system.

0 Karma

twinspop
Influencer

Have you checked the old events? Do their timestamps match their index time? In other words, are they reporting as 2 years old for LOG TIME, but they were actually indexed more recently?

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...