Two of our three search heads have stopped forwarding logs.
It doesn't matter if they are captain or not.
I have verified server.conf and outputs.conf look good using splunk btool server all
and splunk btool outputs all
I have tried cleaning all search heads (remove from cluster, remove non-standard apps and user, run splunk clean all, and put back in the cluster)
The logs are being written, but are not being forwarded to the indexers.
Any ideas what to check next?
I got it figured out.
in /opt/splunk/etc/system/local/inputs.conf
[default]
host =<server>
host was set to the same hostname on all three search heads.
No idea how that happened, but setting host to the respective hostname and cluster restart fixed the problem
I got it figured out.
in /opt/splunk/etc/system/local/inputs.conf
[default]
host =<server>
host was set to the same hostname on all three search heads.
No idea how that happened, but setting host to the respective hostname and cluster restart fixed the problem
Does the outputs.conf have this line
[indexAndForward]
index = false
Ref: http://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Forwardsearchheaddata
Yes. They are using index=false