I have two search heads, which are not clustered, only my indexers are clustered, the search heads are separate.
Both worked fine, but recently I must have misconfigured something (unintenionally obviously), because one of my search heads are not able to send any data to my indexers.
The _internal index doesn't contain any data from my problematic search head, and if I try to write something to a summary index with the command "collect", it also fails.
However, the search head started to create buckets locally to store the _internal index.
I was trying to compare the inputs,outputs.conf files against my working search head, but I haven't found anything.
I'm able to run searches from my problematic one, so it can access the cluster, but can't send any data.
Ensure that your search head is configured to forwarder search head data to indexers, as described in below link.
http://docs.splunk.com/Documentation/Splunk/6.4.2/DistSearch/Forwardsearchheaddata
This is the part where I got lost...
I've queried the running config with btool, and there is no tcpout group configured in my search head (the one which works fine), and there is no
server =
option in the outputs.conf at all.
outputs.conf is the one you need to check. See if there is an additional outputs.conf on the problematic search head that is taking precedence.
You can also verify by running btool command to check what configuration is in effect.
./splunk cmd btool outputs list