Getting Data In

Forwarder tcpout_connections blocked

Chris_R_
Splunk Employee
Splunk Employee

This configuration is two 3.4.2 forwarders -> two 4.1.2 indexers.
Forwarders have two UDP inputs & two seperate assigned sourcetypes on these UDP inputs, props/transforms/outputs entries are doing _TCP_ROUTING to two seperate indexers.
Config seems ok for the most part. However they are getting constantly blocked tcpout_connections messages in metrics.log

splunkd.log Error on the forwarders

07-07-2010 06:11:29.452 WARN  TcpOutputProc - TcpSendThread: Connection to server lost - retrying: Broken pipe  
07-07-2010 06:11:29.452 WARN  TcpOutputProc - Connection dropped by Indexer. Possible version mismatch with indexer. Please check compatibility with indexer version  

splunkd.log errors on the indexer

07-08-2010 01:15:13.501 ERROR TcpInputProc - Error encountered for connection from host=< ip address >, ip=< ip address >. Timeout  
07-08-2010 01:15:13.501 INFO  TcpInputProc - Hostname=< ip address > closed connection  
07-08-2010 01:15:13.501 WARN  PipelineInputChannel - channel source::udp:515|host::192.168.88.25|somesourcetypel|remoteport::41108" ended without a done-key  
07-08-2010 01:15:13.501 WARN  PipelineInputChannel - channel "source::udp:514|host::192.168.8.204|somesourcetypee|remoteport::41108" ended without a done-key  
07-08-2010 01:15:13.501 WARN  PipelineInputChannel - channel "source::udp:515|host::192.168.88.26|somesourcetype|remoteport::41108" ended without a done-key  
07-08-2010 01:15:13.501 WARN  PipelineInputChannel - channel "source::/opt/splunk/var/log/splunk/splunklogger.log|host::NCCForwarder|splunklogger|remoteport::41108" ended without a done-key  
07-08-2010 01:15:13.501 WARN  PipelineInputChannel - channel "source::udp:515|host::192.168.88.27|somesourcetype|remoteport::41108" ended without a done-key  

one other odd entry i see in the inputs.conf of the indexers, seems like this is a older spec file setting to route certain data to queues instead of letting splunk do it automatically?

[splunktcp]
route = has_key:_utf8:indexQueue;has_key:_linebreaker:indexQueue;absent_key:_utf8:parsingQueue;absent_key:_linebreaker:parsingQueue;
Tags (2)
0 Karma

Chris_R_
Splunk Employee
Splunk Employee

GK: These are full forwarders, here's the outputs from a forwarder

[tcpout]  
indexAndForward = false  


[tcpout:stonegateGroup]
disabled = false  
server=10.20.12.35:9001  

[tcpout:fortimailGroup]  
disabled = false  
server=10.20.12.33:9997  

and the inputs.conf from a indexer

[default]  
index = default  
host = fortimailsplunk  
_rcvbuf = 196608  

[monitor://$SPLUNK_HOME/var/spool/splunk]  
move_policy = sinkhole  

[fschange:$SPLUNK_HOME/etc]  
signedaudit = true  
sendEventMaxSize = -1  
recurse = true  
pollPeriod = 600  
filesPerDelay = 10  
delayInMills = 100  
followLinks = false  
fullEvent = false  
hashMaxSize = -1  

[splunktcp]  
route = has_key:_utf8:indexQueue;has_key:_linebreaker:indexQueue;absent_key:_utf8:parsingQueue;absent_key:_linebreaker:parsingQueue;  

Note: I had them remove the tcp route = stanza seems to not be blocking this morning, could be a slower day...but i'll know for sure next week

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

That route is in fact in the etc/system/default/inputs.conf for 4.x machines. Someone might have copied it over. Don't mess with it.

Please clarify if these are heavy forwarders, or LWF's tweaked to collect UDP as well? It would be helpful to see the outputs.conf in the forwarders and the inputs.conf on the indexer.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...