This configuration is two 3.4.2 forwarders -> two 4.1.2 indexers.
Forwarders have two UDP inputs & two seperate assigned sourcetypes on these UDP inputs, props/transforms/outputs entries are doing _TCP_ROUTING to two seperate indexers.
Config seems ok for the most part.
However they are getting constantly blocked tcpout_connections messages in metrics.log
splunkd.log Error on the forwarders
07-07-2010 06:11:29.452 WARN TcpOutputProc - TcpSendThread: Connection to server lost - retrying: Broken pipe
07-07-2010 06:11:29.452 WARN TcpOutputProc - Connection dropped by Indexer. Possible version mismatch with indexer. Please check compatibility with indexer version
splunkd.log errors on the indexer
07-08-2010 01:15:13.501 ERROR TcpInputProc - Error encountered for connection from host=< ip address >, ip=< ip address >. Timeout
07-08-2010 01:15:13.501 INFO TcpInputProc - Hostname=< ip address > closed connection
07-08-2010 01:15:13.501 WARN PipelineInputChannel - channel source::udp:515|host::192.168.88.25|somesourcetypel|remoteport::41108" ended without a done-key
07-08-2010 01:15:13.501 WARN PipelineInputChannel - channel "source::udp:514|host::192.168.8.204|somesourcetypee|remoteport::41108" ended without a done-key
07-08-2010 01:15:13.501 WARN PipelineInputChannel - channel "source::udp:515|host::192.168.88.26|somesourcetype|remoteport::41108" ended without a done-key
07-08-2010 01:15:13.501 WARN PipelineInputChannel - channel "source::/opt/splunk/var/log/splunk/splunklogger.log|host::NCCForwarder|splunklogger|remoteport::41108" ended without a done-key
07-08-2010 01:15:13.501 WARN PipelineInputChannel - channel "source::udp:515|host::192.168.88.27|somesourcetype|remoteport::41108" ended without a done-key
one other odd entry i see in the inputs.conf of the indexers, seems like this is a older spec file setting to route certain data to queues instead of letting splunk do it automatically?
[splunktcp]
route = has_key:_utf8:indexQueue;has_key:_linebreaker:indexQueue;absent_key:_utf8:parsingQueue;absent_key:_linebreaker:parsingQueue;
GK: These are full forwarders, here's the outputs from a forwarder
[tcpout]
indexAndForward = false
[tcpout:stonegateGroup]
disabled = false
server=10.20.12.35:9001
[tcpout:fortimailGroup]
disabled = false
server=10.20.12.33:9997
and the inputs.conf from a indexer
[default]
index = default
host = fortimailsplunk
_rcvbuf = 196608
[monitor://$SPLUNK_HOME/var/spool/splunk]
move_policy = sinkhole
[fschange:$SPLUNK_HOME/etc]
signedaudit = true
sendEventMaxSize = -1
recurse = true
pollPeriod = 600
filesPerDelay = 10
delayInMills = 100
followLinks = false
fullEvent = false
hashMaxSize = -1
[splunktcp]
route = has_key:_utf8:indexQueue;has_key:_linebreaker:indexQueue;absent_key:_utf8:parsingQueue;absent_key:_linebreaker:parsingQueue;
Note: I had them remove the tcp route = stanza seems to not be blocking this morning, could be a slower day...but i'll know for sure next week
That route is in fact in the etc/system/default/inputs.conf for 4.x machines. Someone might have copied it over. Don't mess with it.
Please clarify if these are heavy forwarders, or LWF's tweaked to collect UDP as well? It would be helpful to see the outputs.conf in the forwarders and the inputs.conf on the indexer.