It's always something!
Now my Linux forwarder is saying the following:
05-10-2017 09:11:02.584 -0400 WARN TcpOutputProc - Forwarding to indexer group splunk1 blocked for 87500 seconds.
05-10-2017 09:12:42.594 -0400 WARN TcpOutputProc - Forwarding to indexer group splunk1 blocked for 87600 seconds.
05-10-2017 09:14:22.605 -0400 WARN TcpOutputProc - Forwarding to indexer group splunk1 blocked for 87700 seconds.
I checked to ensure that there was no outputs.conf file on the indexer (we only have 1) which would cause a loop.
I am so lost. Any other ideas?
The log complaining about not reaching the indexer group is from the forwarder?
Can you connect to the one of the indexers from the forwarder manually? You can use openssl to try it, if it can connect just use ctrl+c to break out of it. Don't worry if it mentions ssl handshake failures, this is just to make sure the indexer port is accessible from the forwarder.
openssl s_client -connect {indexer_ip}:9997