Distributed Search: After making a search head also an indexer, why are indexes created on the search head instead found on another indexer?

[howyagoin]

Stupid question time.

I've got a pretty simple setup. Search head, two indexers. Everything works great.

Except that my search head is overly resourced for being a search head, and I'd like to add some indexing to it.

If I go into the Settings and create an Index, I see the directory appear on the Search Head just fine, usual location, but, as soon as I start actually indexing data, by, say, indexing a file or directory, the data appears on one of my Indexers and NOT on the Search Head.

Alpha is the indexer, Beta is an indexer and Gamma is an indexer.

I create the index on Alpha, see it on the file system. I then read the file/data in and assign it to the index, by name, that only exists on Alpha.

However, at the same time, the index is created on Beta, and when the data is read in, it is actually indexed on Beta.

A Splunk search for the data shows the splunk_server as being "beta"...

I have no idea how I wound up setting that up, nor why it's not going to Gamma as well. No clusters have been set up, no replication.

I just want to put Alpha to a bit better use.

What obvious thing am I missing?!

[martin_mueller · Accepted solution]

I'm guessing your search head has been configured to forward to your indexers when setting up the system to store _internal logs where they belong.

[howyagoin]

Well, it hasn't been, at least intentionally. I configured all parts of these and at no time did I specifically tell Alpha to send all data to Beta. Indeed, there is no reference to "beta" anywhere in the configuration files for Alpha other than in the distsearch.conf.

[martin_mueller]

Just to be sure, what do you see when you go to Settings -> Forwarding and receiving -> Configure forwarding on your search head?

[howyagoin]

BINGO. Give the man a gold star, a coffee, a single malt. It apparently has been configured for the IP address of beta, and not beta as a hostname, which is why no grepping would show it. ARGH. This explains things. I'll delete this and see if things go back to what I wanted. Thank you...turns out this was "outputs.conf" and I can't believe I didn't check there.

[jimodonald]

You really can't use the Web UI to create indexes in a distributed index cluster. You need to modify the indexes.conf on each indexer and then restart them.

http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Setupmultipleindexes#Create_and_edit_inde...

[howyagoin]

Interesting, if counter-intuitive. I am certain that in previous versions of Splunk this was not always the case. We used to have a combined indexer/search head, which also had distributed search enabled, and indexes appeared on the local system just fine. This is something "new" which appears to have come up with a recent update.

I'll try this - will have to restart the Search Head to test...will report back. Thanks.

[martin_mueller]

Is this even an indexer cluster or just traditional distributed search?

[howyagoin]

So, I don't mind the change of the post title, ppablo_splunk, but it's not accurate. I don't know that I've made the Search Head an indexer - I WANT to do that, and I want to do it correctly. Your edit isn't accurate, but thanks for trying to tidy up the wording otherwise.