Deployment Architecture

Best practices for hot/warm bucket retention?

splunker969
Communicator

The primary indexers data (Hot+ Warm) data is being full .Please help us in solving this issues . .We are trying to shrink the hot and warm are our primary indexers .
The retention period for hot +warm is 30 days.

What are best practice process? To come out of this issue .

somesoni2
SplunkTrust
SplunkTrust

Data in hot/warm buckets are not managed by Splunk's data retention policy. Data is only frozen (archived or deleted based on what you've configured) from cold bucket directory. If you want to reduce the amount of data you want to retain for your index, ensure that you've configuration to have warm buckets rolled to cold buckets. See this for list of attributes that affect the data buckets:
http://docs.splunk.com/Documentation/Splunk/6.6.3/Indexer/Configureindexstorage#Attributes_that_affe...

Consider (appropriately) updating attributes maxHotSpanSecs, maxDataSize and maxWarmDBCount so that buckets will roll (eventually) to cold bucket.

0 Karma

splunker969
Communicator

Thanks @somesoni2

0 Karma

Sukisen1981
Champion

Correctly explained in detail by @somesoni2 . The data retention policy of 30 days is not a guarantee that ALL data in a 30 day period will be stored in the hot / wam buckets. What is important is to know how much data you are indexing. Things can get a bit complex here. Your organization will pay Splunk for the maximum amount of data indexed in a 24 hour period, but since you are not receiving any license violation warnings, I assume that is not a concern at this point of time.
@somesoni2 says - Consider (appropriately) updating attributes maxHotSpanSecs, maxDataSize and maxWarmDBCount so that buckets will roll (eventually) to cold bucket.
You need to make a decision on what size (how much GB) of your data is needed in hot and warm buckets.

0 Karma

splunker969
Communicator

Thanks @sukisen1981

0 Karma

koshyk
Super Champion

I'm not sure what you meant by Shrink? That's genuine data
Some of the options I can think of are:
1. Reduce 30days hot+warm and move to cold
2. increase disk space for hot storage

0 Karma

splunker969
Communicator

Thanks! koshyk

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...