Getting Data In

Splunk forwarder not sending data - Linux

dustinbrown
New Member

Greetings,

I have 2 servers that suddenly stopped sending data to the indexer. I am struggling to find the root cause. I can telnet to the indexer from the forwarder just fine.

Here is the outputs.conf

[tcpout]
defaultGroup = default
disabled = false

[tcpout:default]
compressed = true
server = 10.x.x.x:9997
sslCertPath = $SPLUNK_HOME/etc/auth/server.pem
sslPassword = $1$wUgcTqWznVA=
sslRootCAPath = $SPLUNK_HOME/etc/auth/cacert.pem
sslVerifyServerCert = false

Here is the inputs.conf

[default]
host = xxxx

[SSL]
password = $1$PK3DT9mO4713
serverCert = /opt/splunk/etc/auth/server.pem
rootCA = /opt/splunk/etc/auth/cacert.pem

I currently have SSL turned off under server.conf

[general]
guid = xxxxx
serverName = xxxxx

[lmpool:auto_generated_pool_download-trial]
description = auto_generated_pool_download-trial
quota = MAX
slaves = *
stack_id = download-trial

[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder

[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free

[lmpool:auto_generated_pool_enterprise]
description = auto_generated_pool_enterprise
quota = MAX
slaves = *
stack_id = enterprise

[license]
active_group = Enterprise

[sslConfig]
enableSplunkdSSL = false
sslKeysfilePassword = $1$eOiFDozCt+53

Other

The strange thing is, I have mimicked configuration from other servers that are forwarding traffic just fine. I have 2 that will not send any. The logs are not full of errors.

I took over splunk just recently so still very new to all of this.

Starting splunk in debug, I notice the following that looks odd.

05-31-2012 18:06:57.353 DEBUG TcpOutputProc - Cannot find any valid descriptors when looking for new indexer.
05-31-2012 18:06:57.353 DEBUG TcpOutputProc - Looking for indexer...
05-31-2012 18:06:57.353 DEBUG TcpOutputProc - Connection not available. Waiting for connection ...

Does any one have any insight??

0 Karma

wrangler2x
Motivator

I am experiencing this with one of my forwarders. I installed the forwarder software, got the thing up-and-running using port 9998, and was taking logs from it on the indexer. After running just fine for a week, the logs just quit coming. On the indexer I see this error in splunkd.log:

10-13-2014 11:08:22.115 -0700 ERROR TcpInputProc - Error encountered for connection from src=xxx.xxx.xxx.xxx:50059. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

There is no connection established for the forwarder on the indexer (using netstat to look for it). Nothing that I know of has changed on either system. Very strange. did you ever find out what caused it?

mloven_splunk
Splunk Employee
Splunk Employee

I'm assuming that the inputs.conf you posted is from your indexer?

If so, I don't see a stanza in your inputs.conf for port 9997.

0 Karma

dustinbrown
New Member

We do have SOS installed and running. I see the following error for one of the servers but not the other

05-31-2012 17:57:24.909 +0000 ERROR TcpInputProc - Error encountered for connection from src=x.x.x.x:36447. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
host=xxxxxx Options| source=/opt/splunk/var/log/splunk/splunkd.log Options| component=TcpInputProc Options| log_level=ERROR Options

0 Karma

sdaniels
Splunk Employee
Splunk Employee

You may want to install the Splunk on Splunk app to help with troubleshooting issues. http://splunk-base.splunk.com/apps/29008/sos-splunk-on-splunk

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...