Deployment Architecture

how is age of an event calculated before the bucket is frozen

pavankumarh
Path Finder

hi, when we set frozenTimePeriodInSecs=30 days for an index. i have read through the documentation and forums that the bucket moves to frozen state when all the data in the given bucket is more than 30 days old. but i need some more clarity. basically the condition i think is one of the below for the latest event in the bucket.

_time > frozenTimePeriodInSecs
OR
_indextime >frozenTimePeriodInSecs

Please let me know which time controls the retention.

0 Karma
1 Solution

lukejadamec
Super Champion
0 Karma

lukejadamec
Super Champion

It's _time
See this post. It provides a very good review:
https://wiki.splunk.com/Deploy:SplunkBucketRetentionTimestampsAndYou

0 Karma

pavankumarh
Path Finder

Thanks Luke.. The explanation in wiki link was a bit complex and took time to understand but it certainly supports your answer.

I was running two parallel daily reports - one based on _time and the other based on _indextime to verify if my retention values were working as expected. But both confusing me.
Now i will concentrate on _time based one for my further analysis.

Thanks Again.

0 Karma

lukejadamec
Super Champion

It is complex, but necessarily so because while you may want old data, you probably most certainly want current data.

What may not be evident in the wiki post is the indexing very old logs and logs 'in the future'.

Let's say for example you have logs that are older than the MAX_DAYS_AGO parameter from props.conf (default 2000 days) - all events older than the MAX_DAYS_AGO will have the _time value of the last acceptable timestamp in that log file, and if all events are older than MAX_DAYS_AGO, then all events will have the current (index time) timestamp for _time.

So, in addition to knowing that Freezing data is based on _time, you must also understand that it is based on the most recent time in the buckets (restart Splunkd = new buckets), and that indexing very old logs may give you event times (_time) that are not what you expected (MAX_DAYS_AGO defaults to 2000 days).

richgalloway
SplunkTrust
SplunkTrust

I'm 95% certain it's _time.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...