Can't add a search peer to a cluster search head

[responsys_cm]

We're building out a new cluster and want to keep our old data accessible. Since there is no way to forward all of our old data to the new cluster, the recommendation of support was to add the old server as a search peer on the cluster search head.

When I go in to the GUI on the search head and go to the distributed search section, the "Add" button isn't there. I can't add the old server to the cluster because the disk and directory structure is too different.

Do I have to remove the search head from the cluster and then just configure distributed search to search the three cluster indexes and the old server?

Thanks.

Craig

[grijhwani]

Not necessarily. You could, in fact, install a second instance of Splunk onto one of the new machines, with storage for the old logs and address it that way. (You would have to set Splunk up with non-standard ports - e.g. 9089 for management.) However, that still does not address your problem directly, because you would have to add the second instance of Splunk as a separate searchable indexer to the search head. But it us something to consider if you are concerned about the longevity of the legacy hardware.

[responsys_cm]

I want to be able to search the data on my old Splunk server and the data on the new cluster. A month from now if I want to run an annual report, I need to be able to search both my old Splunk server's data and the new cluster's data. And I don't want to have to copy my old indexes to one of the cluster indexers. I'd have to rename all of those indexes and then tweak all of the searches that have a "index=xyz" in it...

[bmacias84]

What do you mean? Do you still want your old indexes search able?

[responsys_cm]

Is it possible to make my old Splunk server a search head for the cluster? Will it then be able to search its own local indexes and the remote ones?

[bmacias84]

Currently you are not able to add a none clustered indexer to an to a distributed clustered environment. I have talked to others and the suggested work around is to turn your nonclustered indexers into a cluster of 1. At this point you should be able to added a stand alone indexer as a search peer. I have never done this.

[grijhwani]

Are you sure your Enterprise licence is properly configured and available to the search head?

What is in your $SPLUNK_HOME/etc/system/local/distsearch.conf (or indead default/distsearch.conf)?

What do Splunk's internal logs have to say on the matter? ($SPLUNK_HOME/var/log/splunk/...)