Hi Guiseppe and Jon,
Guys - thanks for your help and detailed answers. In the end, the problem had been user error (mine, of course). So the following are basic troubleshooting steps which can be used to identify any installation/configuration errors, and hopefully save others my head banging of a new install at 3 AM.
1) Insure that the Splunk service is indeed running on the indexer host (no splunk, no communication):
ps aux -l | egrep splunk
sudo splunk status
2) Insure that the port which the indexer host is listening on matches the one which which the forwarders are sending data to (the deaf ear does not listen):
sudo netstat -l
3) Insure that ALL firewalls between the forwarder and listener port are open (threading the needle):
telnet 192.168.x.x 9997
4) Insure that Splunk is running on the forwarder (no quarterback, no pass)
sudo splunk status
5) Insure that the the forwarder is correctly "registered" with the indexer (no voter registration, no vote):
sudo splunk list forward-server
Finally, it's a good check to reboot both indexer and forwarder after installation ("registering" the indexer). Restarting the splunk services on both should be enough, but if sudo splunk enable-bootstart was missed then there will be no Splunk service on restart.
Again, thanks.
David
... View more