Getting Data In

Configuring Distributed Indexes

craigmunro
Path Finder

Hi,

I'm planning to use the deployment server to deploy configs to my indexers and use network storage for the cold DB. I have the following in indexes.conf:

[foo]
homePath = $SPLUNK_DB/foo/db
coldPath = /nfs/foo-splunk/storage/foo/colddb
thawedPath = $SPLUNK_DB/foo/thaweddb
maxDataSize = auto_high_volume
maxWarmDBCount = 7
maxTotalDataSizeMB = 220000
frozenTimePeriodInSecs = 1900800

When I have multiple indexes how is this applied? For example is the maxTotalDataSizeMB per indexer or per index?

Can multiple indexers use the same location for the colddb?

Thanks

Tags (2)

lguinn2
Legend

To clarify, let's say I have two indexers. On each of these indexers, I set the size of foo in indexes.conf as

maxTotalDataSizeMB=100000

Each indexer must write to its own directory (typically on local disk for performance reasons). The max size of the index is the SUM of all of the space allocated by all indexers = 200000 MB

0 Karma

craigmunro
Path Finder

Setting the coldPath as /nfs/$HOSTNAME/foo in indexes.conf works which means I can use the deployment server to configure my indexers.

0 Karma

Genti
Splunk Employee
Splunk Employee

The above stanza will be applied to the foo index only. no matter how many other indexes you have.
maxTotalDataSizeMB will be per index, ie only for the foo index. (if you are deploying this to more then one indexer, this will be so for all those indexers, but again, only for the foo index.)

You can NOT use the same location for multiple indexers (otherwise you would have clashes between the data). If you want to use the same nfs then you should probably create directories for all your indexers say:

/nfs/indexer1/foo/
/nfs/indexer2/foo/
etc..
0 Karma

Genti
Splunk Employee
Splunk Employee
  1. you should NOT have all your indexers read and write files to that exact same folder on the NFS. If you set them up in different directories, then maxTotalDataSizeMB will apply to a single indexer/index. Hence if you say 10 Gig, and then each indexer will have 10Gig max for that index. if you have 10 indexers, each of 10 Gig for the particular index, then you need 100Gigs...
0 Karma

craigmunro
Path Finder

Thanks Genti. I understand that this config only applies to the foo index I'm trying to clarify what happens when this index is distributed across multiple indexers. You say that maxTotalDataSizeMB would apply to all indexers. Does that mean it has the same limit on each indexer or that the limit is across all indexers?

Is the hostname available as a variable in indexes.conf so I can do /nfs/$HOSTNAME/foo/ and deploy the same config to all indexers?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...