I have a fairly basic deployment - one Search Head configured with two distributed search peers/indexers. Each peer is in a different data center, serving as the role of an indexer. I've installed the Universal Forwarder on the hosts in each datacenter, and they are successfully sending Windows Event Log data to their respective indexer. No clustering or replication is taking place, and both indexers have checked into the licensing service (which is also running on my Search Head). Both peers are showing up as "Up" and replication status is "Successful".
On each indexer, I can execute searches against indexes specific to the indexes hosted on each indexer. Put another way (because that sounded really confusing), when I search for data appropriate to each indexer, I receive the expected results.
While logged into the search head, when I search for data specific to data hosted by Indexer01, I receive the expected results. However, when I search for data hosted by Indexer02, I get no results. I've restarted the Splunk services many times, I've removed and re-added the "failing" peer, to no avail. Logs are "clean" in that I'm not finding any glaring errors in both the "splunkd" and "remote_searches" log files.
What am I missing? Can a search head only utilize a single peer? What log should I be looking at?
Thank you in advance!
-Todd
... View more