Reporting

How do I stop old records being deleted?

simonsjw
Engager

I have a set of log records dating from 2009 to 2011. I upload them to Splunk and set MAX_DAYS_AGO=10,000 as well as setting the Tsidx Retention Policy to disabled.

However, whenever I restart splunk, the records are deleted.

Does anyone know how to stop the old records being deleted?

Thanks and regards,

Simon

0 Karma

gjanders
SplunkTrust
SplunkTrust

There are a couple of settings to check here, as per the indexes.conf documentation , you mention 2009 to 2011, the default frozenTimePeriodInSecs is 6 years, so you can either change that for every index under:

[default]

Or per-index.

The other setting to check is your maxTotalDataSizeMB which is set to 500,000MB by default, if that is exceeded you will also have data frozen.

To check what the current sizing is of the indexes refer to the monitoring console of Splunk , it has a panel on index information which will show the frozenTimePeriodInSecs and current oldest data et cetera.

If the data has been frozen already the only restoration method is restore from backup as per the documentation

A search such as (if you have a Linux based indexer):

index=_internal sourcetype=splunkd source="/opt/splunk/var/log/splunk/splunkd.log" "BucketMover - will attempt to freeze"

Will show all bucket freezing activity on the indexers...I have an example query here to detect when data is frozen due to reaching the index size limits but not the frozenTimePeriodInSecs.
To make the above search more efficient you may wish to add a host= clause for your indexer names...

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

MAX_DAYS_AGO is a setting that only affects processing at indexing time. It is not a setting that controls data retention. Refer to the doc link posted below for configuring index retention policies.

0 Karma

schollaert
Explorer

Have a look at the way buckets (records) are rolled to frozen (deleted) https://docs.splunk.com/Documentation/Splunk/7.0.0/Indexer/Setaretirementandarchivingpolicy

Best regards,

Jan

simonsjw
Engager

Thanks so much Jan - I read the docs and can see that I needed to update frozenTimePeriodInSecs in my indexes.conf file.

However, it appears there are many and although I believe I've updated the relevant one, my records are still being deleted.

Appreciate you taking the time - definitely the right track.

Simon

0 Karma
Get Updates on the Splunk Community!

Updated Team Landing Page in Splunk Observability

We’re making some changes to the team landing page in Splunk Observability, based on your feedback. The ...

New! Splunk Observability Search Enhancements for Splunk APM Services/Traces and ...

Regardless of where you are in Splunk Observability, you can search for relevant APM targets including service ...

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...