Knowledge Management

Why are the index retention settings not being applied for my summary index as expected?

rakesh_498115
Motivator

Hi All,

I have a summary index called "my_index", which has the data every 30 min from a saved search. I want this data to be in the index for the span of 30 min from the time of data is loaded in the index. For this, I have used the below settings, but it seems it's not working...

Am I missing anything?

[my_index]
coldPath = $SPLUNK_DB/my_index/colddb
homePath = $SPLUNK_DB/my_index/db
thawedPath = $SPLUNK_DB/cold/my_index/thaweddb
maxHotSpanSecs =1800
frozenTimePeriodInSecs = 1800
rotatePeriodInSecs = 60
repFactor = auto

Can someone pls help me with the configuration for the setting I require?

Many Thanks in advance.

Tags (2)
0 Karma

renjith_nair
SplunkTrust
SplunkTrust

Just out of curiosity, why do you need to remove data after 30 minutes instead of constrain that on search level? As you might be already aware you can restrict the data to 30 minutes in the search.

Coming to your question, if you see at the indexes.conf doc,

maxHotSpanSecs = <positive integer>
    * Upper bound of timespan of hot/warm buckets in seconds.
    * Defaults to 7776000 seconds (90 days).
    * NOTE: If you set this too small, you can get an explosion of hot/warm
      buckets in the filesystem.

    * If you set this parameter to less than 3600, it will be automatically reset to
      3600, which will then activate snapping behavior (see below).
    * This is an advanced parameter that should be set
      with care and understanding of the characteristics of your data.
    * Highest legal value is 4294967295

** If you set this parameter to less than 3600, it will be automatically reset to 3600, which will then activate snapping behavior (see below).*
And

frozenTimePeriodInSecs = <nonnegative integer>
    * Number of seconds after which indexed data rolls to frozen.
    * If you do not specify a coldToFrozenScript, data is deleted when rolled to frozen.
    * IMPORTANT: Every event in the DB must be older than frozenTimePeriodInSecs before it will roll. Then, the DB 
      will be frozen the next time splunkd checks (based on rotatePeriodInSecs attribute).
    * Defaults to 188697600 (6 years).
    * Highest legal value is 4294967295

    * IMPORTANT: Every event in the DB must be older than frozenTimePeriodInSecs before it will roll. Then, the DB 
      will be frozen the next time splunkd checks (based on rotatePeriodInSecs attribute).

So it looks like you are not giving splunk enough time to do it's job 🙂

Happy Splunking!
0 Karma

rakesh_498115
Motivator

Thanks for the update nair,

Actually i have a usecase sceneraio to run the search on the summary index "my_index" for all time , for every 2 hours. So when i my search runs on the "my index" for every 2 hours i want the data in the "my index" to be moved out of index or rolled up .

so using the above configuration but its not working.

Based on your comment i have used the following configuration alone , to retain the data for 1 hour , but doesnt seems give postive results.

[my_index]
coldPath = $SPLUNK_DB/my_index/colddb
homePath = $SPLUNK_DB/my_index/db
thawedPath = $SPLUNK_DB/cold/my_index/thaweddb
maxHotSpanSecs =1800
rotatePeriodInSecs = 60
repFactor = auto

0 Karma

renjith_nair
SplunkTrust
SplunkTrust

First of all, why don't your restrict the search which runs for every 2 hours to run for specific time instead of "All Time"?
I believe it's not advisable to do the rolling in a very small time interval. Moreover, as per doc minimum value of maxHotSpanSecs is 3600 "If you set this parameter to less than 3600, it will be automatically reset to 3600, which will then activate snapping behavior"

Also you have to include the configurations for hot->warm->cold->frozen

Have a look at the following doc https://wiki.splunk.com/Deploy:BucketRotationAndRetention

Happy Splunking!
0 Karma

rakesh_498115
Motivator

Hi Nair,

yes the problem here to restrict search is , the summary index which am using today to caputure the specfic data can be different data at diff intervals for the same timestamp. So leaving the index contents as such would give me a duplicate set of data. So this particular index "my_index" i am using a flag index or temp index to hold data for some time.

Can you share the sample configuration , which does the trick for my configuraion ??

Thanks,

0 Karma

renjith_nair
SplunkTrust
SplunkTrust

Hello Rakesh,

Again I would do the adjustments in the search to get de-duplicated data. However, try below configuration. Please be aware that It's not recommended

[my_index]
coldPath = $SPLUNK_DB/my_index/colddb
homePath = $SPLUNK_DB/my_index/db
thawedPath = $SPLUNK_DB/cold/my_index/thaweddb
maxHotSpanSecs =3600 --->rolls hot to warm after 1 hr
maxWarmDBCount = 0 --> rolls warm to cold (since its 0,it will directly roll to cold)
frozenTimePeriodInSecs = 60 -->cold to frozen
rotatePeriodInSecs = 60

Please try this in a test environment and check the logs to see if it happens . Refer http://docs.splunk.com/Documentation/Splunk/6.1/Admin/indexesconf for variable explanations

Happy Splunking!
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...