Security

Data archiving in splunk

splunker_123
Path Finder

Hi
I understand there are four buckets in which datas are rolled out in splunk before deleting - Hot,Warm,Cold and Frozen
In the document it is said - to delete the data by maximum of ages we have to specify the value in indexes.conf and by default it is 250000MB. Does that mean each and every bucket will hold the data till it reaches 250 GB - Say Hot bucket will hold the data till it reaches 250GB then it transfers to Warm and then Warm will hold it reaches 250 GB and it goes on like that?
OR
is that 250 GB applicable only for Frozen bucket if so on what condition other buckets will roll out the data,Can any one please clarify?

Thanks

Tags (5)
0 Karma
1 Solution

kallu
Communicator

Actually there are 4 stages in which your buckets can be; hot, warm, cold and frozen. Typically there are multiple buckets in each stage. I couldn't find any reference to 250GB default from indexes.conf but maybe if you can provide a link to the docs you were reading someone can explain it. Meanwhile here is some good reading to help calculating how much storage Splunk indexes will need (or how to match Splunk config with storage resources)

View solution in original post

0 Karma

MarioM
Motivator

Actually the default maximum size(maxTotalDataSizeMB) for an index is 500,000MB/500GB.

maxTotalDataSizeMB=index(Hot+Warm+Cold)

Frozen means deleted or exported thus not available to splunk.

0 Karma

kallu
Communicator

Actually there are 4 stages in which your buckets can be; hot, warm, cold and frozen. Typically there are multiple buckets in each stage. I couldn't find any reference to 250GB default from indexes.conf but maybe if you can provide a link to the docs you were reading someone can explain it. Meanwhile here is some good reading to help calculating how much storage Splunk indexes will need (or how to match Splunk config with storage resources)

0 Karma

kallu
Communicator

As it says in docs, if/when your index reaches maxTotalDataSize (that is total size of hot, warm and cold data) Splunk will delete data starting from the oldest. The catch is, your index may never reach this size as there are other controls that will freeze your data (ie delete it) before the total size goes up to maxTotalDataSize.

0 Karma

splunker_123
Path Finder

Thank for the reply

http://docs.splunk.com/Documentation/Splunk/latest/admin/Setaretirementandarchivingpolicy

If you look at the above link it says maxTotalDataSize MB=250000 - so my question was ,will the indexer hold the data till it reaches the value specified in maxTotalDataSize? (I'm yet to read the link you've given me)

0 Karma

splunker_123
Path Finder

Any one know about this please?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...