We have about a 3 TB/day ingest rate, spread across about 20 indexes, and we have a 2 to 5 year retention time depending on the index. 4 of these indexes account for 90% of the data, and we are expecting to approximately double or triple our ingest rate eventually. We currently have out maxHotSpanSecs = 86400, maxHotBuckets = 10, and quarantine(Past|Future)Secs = 86400. I have some questions regarding the best practices for setting up buckets and wanted to get feedback on our current settings.
http://dev.splunk.com/view/java-sdk/SP-CAAAEJ2 answers part of 3/4/5 I guess, the regular bucket mechanism applies for Quars as well (so it works as with regular hot/warm buckets):
maxHotBuckets A number that indicates
the maximum number of hot buckets that
can exist per index. When this value
is exceeded, Splunk Enterprise rolls
the least recently used (LRU) hot
bucket to warm. Both normal hot
buckets and quarantined hot buckets
count towards this total. This setting
operates independently of
"maxHotIdleSecs", which can also cause
hot buckets to roll.
So consider to raise maxhotbuckets with 1, or way better: prevent or filter out those 'bad timestamp' events with props.conf on your forwarder so Quar is not used anymore, as Quar takes up one slot. Using IndexedTime may be a good first step.
quarantinePastSecs = 86400 seems rather low in comparison to the splunk default of 900 days. When adding a new logfile/source it'll end up automatically in a Quar bucket with all older >1 day events. Or when a weekly batch file is run, and events are imported on specific dates in the previous week, they end up in Quar. Is that really necessary?
I wonder how you could set different bucketsettings for quarantined buckets, as I understood settings apply to both normal/quarantined hot buckets?
For 1 and 2 keep the Indexes,conf in mind - are you mostly interested in a good indexing performance, and/or in having a good performance in dashboards or queries done on the imported data? As usual 'it depends' on your specific needs and load.
Compare it to looking for something in your drawers; the more drawers you have to check the longer it'll take.
But to store everything in your drawers, you might need more drawers, OR as many drawers but larger ones.