Getting Data In

Automatically source is getting deleted after 24 hours

sravan2j
Explorer

I added source file (.csv file) to splunk using below command,

./splunk add oneshot /root/project/2003.csv –sourcetype sfpd

I can see that 1,50,902 events got indexed.

But exactly after one day, all indexed data from this source file will get deleted except one line (i.e., header of .csv).

I haven't executed delete command. Also I removed the privileges of using delete command, so no one can use it. But still this issue is happening daily.

I am not able to find the solution for this issue.

Please someone help me. Thanks for your help.

Tags (3)
0 Karma
1 Solution

kristian_kolb
Ultra Champion

You've identified the problem:

"the data is 10 years old".

The default retention period that you see in frozenTimePeriodInSecs is about 6 years. That means that as soon as splunk gets time time make the comparison, which in your case is when the hot bucket rolls to warm, it will correctly see that the data should be deleted, and does so.

The solution is to increase the value for frozenTimePeriodInSecs to a higher value, e.g. 400000000 or 500000000, which is about 12 and 15 years, respectively. The highest possible value is 4294967295, which is more than a hundred years...

You can read more about data retention here:

http://docs.splunk.com/Documentation/Splunk/6.0/Indexer/Setaretirementandarchivingpolicy

Hope this helps,

/K

View solution in original post

kristian_kolb
Ultra Champion

You've identified the problem:

"the data is 10 years old".

The default retention period that you see in frozenTimePeriodInSecs is about 6 years. That means that as soon as splunk gets time time make the comparison, which in your case is when the hot bucket rolls to warm, it will correctly see that the data should be deleted, and does so.

The solution is to increase the value for frozenTimePeriodInSecs to a higher value, e.g. 400000000 or 500000000, which is about 12 and 15 years, respectively. The highest possible value is 4294967295, which is more than a hundred years...

You can read more about data retention here:

http://docs.splunk.com/Documentation/Splunk/6.0/Indexer/Setaretirementandarchivingpolicy

Hope this helps,

/K

sravan2j
Explorer

I modified the frozenTimePeriodInSecs to set 400000000 as its value. If I face this issue again, I will message here. Also I want to let you know that, I modified "maxHotIdleSecs" value from 86400 to 604800. Thanking everyone.

0 Karma

sravan2j
Explorer

The following attribute - maxHotIdleSecs in Indexes.conf file has the value 86400. Is this is the reason for this issue??

0 Karma

sravan2j
Explorer

I checked indexes.conf -> FrozenTimePeriodInSecs attribute. Its value is 188697600.

I also ran the following command - "search yourdata | table _time,_raw" as you suggested. The _time value matched with the time in _raw string. Time stamp for the data is 2003-12-01. As the data is 10 years old, may be data is getting deleted. Is it is true? then in that case how I can resolve this issue. Please let me know

lukejadamec
Super Champion

What somesoni2 said, and when the data is searchable check the timestamp of the data:
search yourdata | table _time,_raw
The _time value should match the time in the _raw string, and both should make sense.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

A good idea will be to check the splunk data retention period for the index where this source's data is stored. Indexer.conf-> FrozenTimePeriodInSecs attribute. If this attribute exists for your index and its value is 86400, this is the problem. Increase the value to required period in second, and restart the splunk instance.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...