Getting Data In

2 clusters vs clustered and unclustered vs etc/system/local

richkappler
Path Finder

We are running a large multi-site clustered indexer environment which is maturing causing us to make some changes to our hot/warm/cold rollover scheme. The one issue we have is 2 small sites have a different hardware setup than the rest of the environment. Due to this, I can't use the same indexes.conf on these 2 smaller sites that I use in the rest of the indexers.

The question then is what is the best approach to handling this situation? As I see it, I have 3 choices:
1. Run 2 clusters, which would force me to add another clustermaster.
2. Run the 2 smaller sites unclustered. My gut tells me this would be undesirable, but I'd like something a little more concrete than my gut.
3. Put an indexes.conf in etc/system/local for the smaller sites to override the indexes.conf we have in our slave-apps dir for the clustered indexers.

I believe option 3 to be the best but wanted to reach out for some verification and potential alternative suggestions.

0 Karma
1 Solution

micahkemp
Champion

I'm going to make some assumptions that aren't necessarily stated in the original question:

a) You want to make a change to a configuration in indexes.conf in system/local to surgically make one or more changes to the version pushed from the cluster master.
b) This specific change isn't one that will impact the indexer's ability to be a valid member of the cluster ( [volume:] is an example of a parameter that can be different across indexers, and should be fine to set in system/local).

You can make changes to a clustered indexer in system/local to effect a change specific to only that indexer. Best practice warns against making any changes in system/local, and you can indeed make that change in apps/local according to the Precedence for cluster peer nodes documentation. Note that slave-apps/default takes precedences over apps/default, so it would have to be in apps/local to override slave-apps/local.

So, if you need some indexers to use different volume settings than others, I think this would be a valid method of running disparate configurations on clustered indexers.

View solution in original post

micahkemp
Champion

I'm going to make some assumptions that aren't necessarily stated in the original question:

a) You want to make a change to a configuration in indexes.conf in system/local to surgically make one or more changes to the version pushed from the cluster master.
b) This specific change isn't one that will impact the indexer's ability to be a valid member of the cluster ( [volume:] is an example of a parameter that can be different across indexers, and should be fine to set in system/local).

You can make changes to a clustered indexer in system/local to effect a change specific to only that indexer. Best practice warns against making any changes in system/local, and you can indeed make that change in apps/local according to the Precedence for cluster peer nodes documentation. Note that slave-apps/default takes precedences over apps/default, so it would have to be in apps/local to override slave-apps/local.

So, if you need some indexers to use different volume settings than others, I think this would be a valid method of running disparate configurations on clustered indexers.

nickhills
Ultra Champion

I think you are right, in that you can override indexes locally - volume is a good example of where you might wish to do this, however whilst you can specify that a given index exists on different disks, you cant specify different sizes of indexes, which I think is what the question is getting at.

If my comment helps, please give it a thumbs up!
0 Karma

micahkemp
Champion

I agree the initial question was framed inquiring about site-specific retention, which I didn't answer. I made some potentially invalid assumptions about the "question behind the question."

0 Karma

nickhills
Ultra Champion

Can I just check - when you say multi site, you mean 'Splunk Multi-site Cluster', rather than 1 cluster, simply running across multiple sites 🙂
I am assuming the answer is yes..

Have you configured site replication to unburden the small sites with multiple copies of data from the larger sites - this is not an answer to your specific question, but it might buy you some time.

On a previous deployment, I discovered that whilst you can not set a site to have 0 copies of another sites data, you can 'imply' it:
Assuming 5 sites: 3 big ones (s1, s2, s3), and two little (s4, s5):

site_replication_factor = origin:2, site1:2, site2:2, site3:1 total:5

Will mean that your small sites wont get replicated copies of sites 1-3's data, but they will always hold two copies of anything indexed locally. I cant find anything to say this is an officially supported approach, but the documentation does say:

..., the replication factor ... is not a
requirement. An explicit site is a
site that the replication factor
explicitly specifies. A non-explicit
site is a site that the replication
factor does not explicitly specify.

Which I read as "meh..it should be ok"

edit: added link https://docs.splunk.com/Documentation/Splunk/7.0.1/Indexer/Multisitearchitecture

I'm not sure option 3 would work, and I very much doubt that it would be supported - A cluster works on the idea that an entire indexes data exists in full in at least 1 other location - If that location has differing bucket life-cycles, then your remote site is not upholding its end of the bargain.

For the pain and problems you could otherwise be facing, could I offer option 4, which would read 'add more disks' ?
I know that is something you have probably deliberately omitted, but I cant help but think it would be your path of least resistance (until you submit a PO, anyway)

If my comment helps, please give it a thumbs up!
0 Karma

richkappler
Path Finder

Add more disks is not an option.

0 Karma

richkappler
Path Finder

I forgot to put the specific change, need for change here: the 2 smaller sites in question have smaller hotwarm disks, so the change would be hotwarm to cold rollover based on size (maxVolumeDataSizeMB).

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...