Deployment Architecture

Best practices for hot/warm bucket retention?

splunker969
Communicator

The primary indexers data (Hot+ Warm) data is being full .Please help us in solving this issues . .We are trying to shrink the hot and warm are our primary indexers .
The retention period for hot +warm is 30 days.

What are best practice process? To come out of this issue .

somesoni2
SplunkTrust
SplunkTrust

Data in hot/warm buckets are not managed by Splunk's data retention policy. Data is only frozen (archived or deleted based on what you've configured) from cold bucket directory. If you want to reduce the amount of data you want to retain for your index, ensure that you've configuration to have warm buckets rolled to cold buckets. See this for list of attributes that affect the data buckets:
http://docs.splunk.com/Documentation/Splunk/6.6.3/Indexer/Configureindexstorage#Attributes_that_affe...

Consider (appropriately) updating attributes maxHotSpanSecs, maxDataSize and maxWarmDBCount so that buckets will roll (eventually) to cold bucket.

0 Karma

splunker969
Communicator

Thanks @somesoni2

0 Karma

Sukisen1981
Champion

Correctly explained in detail by @somesoni2 . The data retention policy of 30 days is not a guarantee that ALL data in a 30 day period will be stored in the hot / wam buckets. What is important is to know how much data you are indexing. Things can get a bit complex here. Your organization will pay Splunk for the maximum amount of data indexed in a 24 hour period, but since you are not receiving any license violation warnings, I assume that is not a concern at this point of time.
@somesoni2 says - Consider (appropriately) updating attributes maxHotSpanSecs, maxDataSize and maxWarmDBCount so that buckets will roll (eventually) to cold bucket.
You need to make a decision on what size (how much GB) of your data is needed in hot and warm buckets.

0 Karma

splunker969
Communicator

Thanks @sukisen1981

0 Karma

koshyk
Super Champion

I'm not sure what you meant by Shrink? That's genuine data
Some of the options I can think of are:
1. Reduce 30days hot+warm and move to cold
2. increase disk space for hot storage

0 Karma

splunker969
Communicator

Thanks! koshyk

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...