I'm finding the instructions a little confusing but my understanding is I can have Splunk Enterprise on server 1 which is the indexer and the web interface etc.
Server 2 has the forwarder, so I can set up logs that are monitored and they are forwarded so they can be displayed on server 2. ]
Server 1 according to netstat has 9997 established from the forwarder and the forwarder is also showing the same thing so it looks like the network connectivity between my two servers are working fine.
However, in the log file I am receiving the following:
03-19-2017 15:39:00.074 +0000 WARN TcpOutputFd - Connect to 192.168.172.212:9997 failed. Connection refused
03-19-2017 15:39:00.074 +0000 ERROR TcpOutputFd - Connection to host=192.168.172.212:9997 failed
03-19-2017 15:39:01.907 +0000 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
03-19-2017 15:39:12.458 +0000 WARN HttpListener - Socket error from 192.168.172.212 while idling: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
03-19-2017 15:39:13.907 +0000 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
In /opt/splunkforwarder/etc/system/local/outputs.conf I have the following content:
[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = 192.168.172.212:9997
#[tcpout-server://192.168.172.212:9997]
[monitor:/var/log/httpd/access_log]
server = 192.168.172.212:9997
disabled = 0
[monitor:/var/log/httpd/error_log]
server = 192.168.172.212:9997
disabled = 0
Both servers are running CentOS 7 x64
I'm not sure what I'm doing wrong so any help would be greatly appreciated.
I think I've made some progress, I think I had a forwarder added to the indexer the wrong way round so it was effectively forwarding to itself and failing.
In the indexer log, I am now seeing the following
03-19-2017 16:23:28.440 +0000 ERROR TcpInputProc - Message rejected. Received unexpected message of size=369295617 bytes from src=192.168.166.56:33078 in streaming mode. Maximum message size allowed=67108864. (::) Possible invalid source sending data to splunktcp port or valid source sending unsupported payload.
And on the forwarder I am seeing the following:
03-19-2017 16:28:27.099 +0000 WARN HttpPubSubConnection - Unable to parse message from PubSubSvr:
03-19-2017 16:28:27.099 +0000 INFO HttpPubSubConnection - Could not obtain connection, will retry after=65.832 seconds.
03-19-2017 16:28:33.324 +0000 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
03-19-2017 16:28:33.324 +0000 INFO DC:PhonehomeThread - Attempted handshake 30 times. Will try to re-subscribe to handshake reply
03-19-2017 16:28:45.325 +0000 INFO DC:DeploymentClient - channel=tenantService/handshake Will retry sending handshake message to DS; err=not_connected
... View more