Getting Data In

How to configure indexes.conf to keep large volume of data?

balachandar
Engager

Hi,

We have cluster indexer setup with 5 indexers on separate ESX Servers each with 12TB HDD and 128GB RAM.
The cluster replication factor(RF) is 2 and Search factor(SF) is 1. We have one Job scheduler and search head and forwarder nodes.
Splunk version 6.5.3

Our daily log volume is close to 1TB/day. Out of 1TB log volume 85% of data goes to abc_transaction indexes (RAW data).
In our indexes.conf file we have set frozenTimePeriodInSecs = 3888000 (45 days) for abc_transaction RAW index.
Even though we have set frozenTimePeriodInSecs as 45 days, it is able to search only last 10 days of abc_transaction RAW data.

So to keep the abc_transaction data available for 45 days, we have gone through this Splunk sizing link and added maxVolumeDataSizeMB = 5976884(5.7TB) attribute for abc_transaction index.

https://splunk-sizing.appspot.com/#ar=0&c=1&cr=30&ds=4096&hwr=60&i=5&rf=2&rl=10,01&sf=1&v=1000

indexes.conf

[abc_transaction]
repFactor=auto
coldPath = $SPLUNK_DB/abc_transaction/colddb
homePath = $SPLUNK_DB/abc_transaction/db
thawedPath = $SPLUNK_DB/abc_transaction/thaweddb
maxMemMB = 20
maxConcurrentOptimizes = 6
maxHotIdleSecs = 86400
maxHotBuckets = 10
maxDataSize = auto_high_volume
frozenTimePeriodInSecs = 3888000
maxVolumeDataSizeMB = 5976884

The following are my questions:

  1. Which one do I have to use, maxVolumeDataSizeMB or maxTotalDataSizeMB inside the abc_transaction index.
  2. If the customer wants to keep 90 days of abc_transaction data, as per the Splunk sizing link which I mentioned above the maxVolumeDataSizeMB = 11.4TB which is almost the disk size of the indexer, is it possible to set the entire disk size as maxVolumeDataSizeMB?

Any help would be greatly appreciated.

Thanks
Bala.

0 Karma

davpx
Communicator

Your estimation of 5.7T is pretty close. I'd include a little extra room just in case and let your frozen time period take care of trimming the index. maxTotalDataSizeMB is the right one to use here as you don't seem to be using volume management on this index.

0 Karma

ddrillic
Ultra Champion

@balachandar, the big twin is -

maxTotalDataSizeMB = 1400000
frozenTimePeriodInSecs= 7776000

Retention, in this case, is 3 months and total size at around 1.4 TBs.

balachandar
Engager

@ddrillic, thanks.

May i know how do you calculated maxTotalDataSizeMB = 1400000 for the index abc_transaction

0 Karma

ddrillic
Ultra Champion

oh, that was just my index. We just need to calculate it with any of the online calculators...

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...