Why is the retention policy not working on certain indexes (to delete indexed data that are older then 1 year)?

[Hemnaath]

Hi All, Currently we facing a storage issue in one of the indexer instances, though the retention policy has set for an year and it works for most of the indexes and only for few of the indexes we could see the earliest event date Dec 13, 2013 9:44:29 PM latest event date Mar 21, 2017 10:30:02 AM .

We have executed the splunk btool command but not sure what exactly we need to search in that ? we have checked the splunkd.log but unable to find anything related to this index and also we have executed the dbinspect search command with the particular index with time frame set as last 7 days and got some details but not sure what we need to validate from the result.

When checked the buckets hotdb, colddb, datamodel_summary, summary thawed db, We could see very huge amount of data being stored in hot and cold buckets. And also I could see the below db in both Hot and cold buckets and their size was in KB.

drwx--x--x 3 splunk splunk  4096 Oct 21  2014 inflight-db_1406595487_1406595487_10374
drwx--x--x 3 splunk splunk  4096 Oct 22  2014 inflight-db_1408598841_1408598841_10381

Below are the settings configured in indexes.conf file with retention policy set to Global. Splunk version 6.2.1

[volume:Hot]
path = /slogs

[volume:Cold]
path = /slogs

[volume:Base]
path = /slogs

[default]
frozenTimePeriodInSecs = 31536000
enableTsidxReduction = true
timePeriodInSecBeforeTsidxReduction = 7884000

Kindly guide me how to fix this issue, as we are running short of disk space.

thanks in advance.

[somesoni2]

Based on consistency of timestamp on your data, there may be cases where you get a very old timestamp, say Dec 2013, today (may be bug, wrong logging or timestamp parsing). A data bucket is frozen only when the latest event (highest timestamp) on the bucket is older than your retention period. If the old data was received recently it'll be stored in a bucket with latest event within retention period and will be roll to frozen. All Splunk queries/report dashboard will show the earliest timestamp on the index as Dec 2013, even though your retention is 1 year only.

My suggestion would be to also enforce your data retention based on total index size (maxTotalDataSizeMB) along with retention period (frozenTimePeriodInSecs). This way you can start rolling data bucket to frozen before you run out of space. See this for more details.

https://docs.splunk.com/Documentation/Splunk/6.5.2/Indexer/Setaretirementandarchivingpolicy#Freeze_d...

[Hemnaath]

Hi Somesoni2 thanks for your comments, but by setting the maxTotalDataSizeMB = value will it delete the data that are less then a year old ? because we wanted to have data in splunk for an year.

thanks in advance.

[somesoni2]

It may. The retention works on whatever happens first, your bucket ('s latest event) is older than frozenTimePeriodInSecs OR total index size is greater than maxTotalDataSizeMB (default is 500,000 GB). My guess is that in your guess you'll have to increase the disk size on the indexer having problem. Also, you should try to correct the data at the source so that they don't log intermittent old timestamps. If the issue is limited to few sourcetypes, you could consider reducing MAX_DAYS_AGO for such sourcetypes (props.conf on indexer/heavy forwarder)

MAX_DAYS_AGO = <integer>
* Specifies the maximum number of days in the past, from the current date as
  provided by input layer(For e.g. forwarder current time, or modtime for files),
  that an extracted date can be valid. Splunk still indexes events with dates
  older than MAX_DAYS_AGO with the timestamp of the last acceptable event. If no
  such acceptable event exists, new events with timestamps older than
  MAX_DAYS_AGO will use the current timestamp.
* For example, if MAX_DAYS_AGO = 10, Splunk applies the timestamp of the last
  acceptable event to events with extracted timestamps older than 10 days in the
  past. If no acceptable event exists, Splunk applies the current timestamp.
* Defaults to 2000 (days), maximum 10951.

[Hemnaath]

thanks somesonia2 for guiding on this issue, we have added frozenTimePeriodInSecs = 15552000 (6 months) to the individual indexes that we wanted to push the data which are older than 6 months. We have also kept the frozenTimePeriodInSecs = 31536000 ( 1 year) as global setting for all the indexes in indexes.conf and after doing this changes, we got some space.

But still we have some more indexes occupying 1 Terra bit of data and found that they are not in use anymore. So in this case is it good to remove indexes directly using rm -rf indexes or is there any other way to remove the indexes from the mount point using splunk command.

thanks in advance.