We have single instance, which is on the private subnet and it works really well for all the instances that use its private ip. Then we have created a public ELB, but it is giving the connection refused error:
02-18-2016 18:16:15.628 +0000 INFO TcpOutputProc - Connection to x.164.x.254.1:9997 closed. Connection closed by server. 02-18-2016 18:16:15.628 +0000 WARN TcpOutputProc
- Applying quarantine to ip=x.164.x.254.1 port=9997
_numberOfFailures=5 02-18-2016 18:16:15.631 +0000 INFO TcpOutputProc
- Connected to idx=x.23.x.228:9997 02-18-2016 18:16:15.859 +0000 INFO TcpOutputProc - Connected to idx=x.23.x.228:9997 02-18-2016 18:16:15.859 +0000 INFO TcpOutputProc
- Connected to idx=x.23.x.228:9997 02-18-2016 18:16:15.860 +0000 INFO TcpOutputProc - Connection to x.23.x.228:9997 closed. Connection closed by server. 02-18-2016 18:16:15.860 +0000 WARN TcpOutputProc
- Applying quarantine to ip=x.23.x.228 port=9997 _numberOfFailures=5 02-18-2016 18:16:15.860 +0000 INFO TcpOutputProc - Connection to x.23.x.228:9997 closed. Connection closed by server. 02-18-2016 18:16:45.626 +0000 INFO TcpOutputProc
- Removing quarantine from idx=x.164.x.254.1:9997 02-18-2016 18:16:45.629 +0000 INFO TcpOutputProc
- Connected to idx=x.164.x.254.1:9997 02-18-2016 18:16:45.630 +0000 INFO TcpOutputProc - Connection to x.164.x.254.1:9997 closed. Connection closed by server. 02-18-2016 18:16:45.630 +0000 WARN TcpOutputProc
- Applying quarantine to ip=x.164.x.254.1 port=9997
_numberOfFailures=6 02-18-2016 18:16:45.632 +0000 INFO TcpOutputProc
- Connected to idx=x.164.x.254.1:9997 02-18-2016 18:17:15.627 +0000 INFO TcpOutputProc - Removing quarantine from idx=x.23.x.228:9997 02-18-2016 18:17:15.627 +0000 INFO TcpOutputProc
- Removing quarantine from idx=x.164.x.254.1:9997 02-18-2016 18:17:15.630 +0000 INFO TcpOutputProc
- Connected to idx=x.23.x.228:9997 02-18-2016 18:17:15.631 +0000 INFO TcpOutputProc - Connection to x.23.x.228:9997 closed. Connection closed by server. 02-18-2016 18:17:15.631 +0000 WARN TcpOutputProc
- Applying quarantine to ip=x.23.x.228 port=9997 _numberOfFailures=7 02-18-2016 18:17:15.633 +0000 INFO TcpOutputProc - Connected to idx=x.23.x.228:9997 02-18-2016 18:17:15.634 +0000 INFO TcpOutputProc
- Connection to x.23.x.228:9997 closed. Connection closed by server. 02-18-2016 18:17:15.637 +0000 INFO TcpOutputProc - Connected to idx=x.164.x.254.1:9997
Steps taken so far:
- check the firewall ---------> OK
- telnet to the receiver(indexer) on port 9997 ----------> Successfully connected
my outputs.conf for reference:
[tcpout]
defaultGroup = my_splunk
[tcpout:my_splunk]
autoLB = true
server = mysplunk.test.com:9997
Thanks,
HI,
I had the same issue.
The splunk server log showed some SSL3 issues.
"routines:SSL3_GET_RECORD:wrong version number" as this was a brand new installation of the forwarder, I had to search Mr.Google for quite a while to find the right answer.
The only kind of workaround I could find was to enable the insecure SSL-Version.
https://answers.splunk.com/answers/552516/ssl-error-after-upgrading-from-661-to-662.html
Is encryption taking place anywhere? If so, take a look at these and see if they help:
https://answers.splunk.com/answers/28040/forwarder-tcpoutputproc-connection-closed-by-server.html
https://answers.splunk.com/answers/112014/ssl-connection-between-indexer-and-forwarder-validation.ht...
Also, if you're using SSL/TLS, some firewalls are smart enough to require the protocol be set in the rules.
What does your ELB configuration look like? What is your keepalive set to? Keep in mind you need to set the protocol to TCP, this is not an HTTP protocol you are load balancing.
Generally speaking, Splunk advises AGAINST putting a load balancer in front of your indexers.
http://docs.splunk.com/Documentation/Splunk/6.1/Forwarding/Setuploadbalancingd
Note: When implementing load balancing between forwarders and receivers, you must use the forwarder's inherent capability. Do not use an external load balancer. The use of external load balancers between forwarders and receivers will not work properly.