My end server is not visible in search and i see the below errors in the log.
TcpOutputProc - the 'defaultGroup' property contains an invalid group name - heavy_forwarder
[tcpout]
connectionTimeout = 45
defaultGroup = all_indexers
indexAndForward = 0
forwardedindex.0.whitelist = .*
[tcpout:all_indexers]
autoLB = true
server = heavyforwarder.hostname:9997
04-08-2015 15:53:57.351 -0700 INFO IntrospectionGenerator:disk_objects - Enabled: indexes|volumes|dispatch=true fishbucket=true partitions=true
04-08-2015 15:53:57.351 -0700 INFO IntrospectionGenerator:disk_objects - I-data gathering (Disk Objects) starting; period=600s
04-08-2015 15:53:57.357 -0700 INFO HttpPubSubConnection - SSL connection with id: connection_172.29.26.66_8089_n01scl001.aap.csaa.pri_n01scl001_77D5E0FB-3193-4598-9A96-2449BB33930A
04-08-2015 15:53:57.371 -0700 WARN DistributedPeerManager - feature=DistSearch not enabled for your license level
04-08-2015 15:53:57.371 -0700 INFO IndexProcessor - running splunkd specific init
04-08-2015 15:53:57.371 -0700 INFO loader - Initializing from configuration
04-08-2015 15:53:57.372 -0700 INFO PipelineComponent - registering timer callback name=triggerCollection callback=0xbb6320 arg=0x7f1a8282e180
04-08-2015 15:53:57.385 -0700 INFO PipelineComponent - registering timer callback name=triggerCollection callback=0xbb6320 arg=0x7f1a8202e180
04-08-2015 15:53:57.385 -0700 INFO TcpOutputProc - Initializing with fwdtype=lwf
04-08-2015 15:53:57.385 -0700 INFO ServerRoles - Declared role=lightweight_forwarder.
04-08-2015 15:53:57.390 -0700 INFO TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : forwardedindex.0.whitelist
04-08-2015 15:53:57.390 -0700 INFO TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : forwardedindex.1.blacklist
04-08-2015 15:53:57.390 -0700 INFO TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : forwardedindex.2.whitelist
04-08-2015 15:53:57.390 -0700 INFO TcpOutputProc - Initializing connection for non-ssl forwarding to n01sml105.aap.csaa.pri:9997
04-08-2015 15:53:57.390 -0700 INFO TcpOutputProc - tcpout group all_indexers using Auto load balanced forwarding
04-08-2015 15:53:57.390 -0700 INFO TcpOutputProc - Group all_indexers initialized with maxQueueSize=512000 in bytes.
04-08-2015 15:53:57.403 -0700 INFO TcpOutputProc - Initializing connection for non-ssl forwarding to n01sml105.aap.csaa.pri:9997
04-08-2015 15:53:57.403 -0700 INFO TcpOutputProc - tcpout group heavy_forwarders using Auto load balanced forwarding
04-08-2015 15:53:57.403 -0700 INFO TcpOutputProc - Group heavy_forwarders initialized with maxQueueSize=512000 in bytes.
04-08-2015 15:53:57.403 -0700 ERROR TcpOutputProc - the 'defaultGroup' property contains an invalid group name - heavy_forwarder - skipping
04-08-2015 15:53:57.403 -0700 WARN TcpOutputProc - Default groupList has not yet been calculated!
04-08-2015 15:53:57.403 -0700 INFO PipelineComponent - Pipeline merging disabled in default-mode.conf file
04-08-2015 15:53:57.403 -0700 INFO PipelineComponent - Pipeline typing disabled in default-mode.conf file
04-08-2015 15:53:57.403 -0700 INFO PipelineComponent - Pipeline vix disabled in default-mode.conf file
04-08-2015 15:53:57.430 -0700 WARN HttpPubSubConnection - Received message for an unsubscribed channel: deploymentServer/phoneHome/default/reply/n01scl001/77D5E0FB-3193-4598-9A96-2449BB33930A
04-08-2015 15:53:57.440 -0700 INFO PipelineComponent - registering timer callback name=triggerCollection callback=0xbb6320 arg=0x7f1a81045180
04-08-2015 15:53:57.440 -0700 INFO TcpInputConfig - SSL clause not found or servercert not provided - SSL ports will not be available
04-08-2015 15:53:57.441 -0700 INFO TcpInputProc - Registering metrics callback for: tcpin_connections
04-08-2015 15:53:57.441 -0700 INFO PipelineComponent - registering timer callback name=triggerCollection callback=0xbb6320 arg=0x7f1a88c5fc80
04-08-2015 15:53:57.491 -0700 INFO PipelineComponent - registering timer callback name=triggerCollection callback=0xbb6320 arg=0x7f1a87c56180
04-08-2015 15:53:57.491 -0700 INFO PipelineComponent - registering timer callback name=triggerCollection callback=0xbb6320 arg=0x7f1a87c56480
04-08-2015 15:53:57.551 -0700 INFO PipelineComponent - registering timer callback name=triggerCollection callback=0xbb6320 arg=0x7f1a87c56600
04-08-2015 15:53:57.551 -0700 INFO PipelineComponent - Pipeline fifo disabled in default-mode.conf file
04-08-2015 15:53:57.551 -0700 INFO PipelineComponent - Launching the pipelines.
04-08-2015 15:53:57.552 -0700 INFO loader - Limiting REST HTTP server to 5461 sockets
04-08-2015 15:53:57.553 -0700 INFO loader - Limiting REST HTTP server to 1299 threads
04-08-2015 15:53:57.597 -0700 INFO TailingProcessor - TailWatcher initializing...
04-08-2015 15:53:57.597 -0700 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
04-08-2015 15:53:57.598 -0700 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
04-08-2015 15:53:57.598 -0700 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
04-08-2015 15:53:57.598 -0700 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
04-08-2015 15:53:57.598 -0700 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
04-08-2015 15:53:57.598 -0700 INFO TailingProcessor - Parsing configuration stanza: monitor:///opt/elasticsearch/apache-tomcat-7.0.53/logs/catalina.out.
04-08-2015 15:53:57.598 -0700 INFO TailingProcessor - Adding watch on path: /opt/elasticsearch/apache-tomcat-7.0.53/logs/catalina.out.
04-08-2015 15:53:57.598 -0700 INFO TailingProcessor - Adding watch on path: /opt/splunk/splunkforwarder/etc/splunk.version.
04-08-2015 15:53:57.598 -0700 INFO TailingProcessor - Adding watch on path: /opt/splunk/splunkforwarder/var/log/splunk.
04-08-2015 15:53:57.598 -0700 INFO TailingProcessor - Adding watch on path: /opt/splunk/splunkforwarder/var/spool/splunk.
04-08-2015 15:53:57.598 -0700 INFO BatchReader - State transitioning from 2 to 0 (initOrResume).
04-08-2015 15:53:57.608 -0700 INFO TcpOutputProc - Connected to idx=172.29.28.72:9997
04-08-2015 15:53:57.608 -0700 INFO TcpOutputProc - Connected to idx=172.29.28.72:9997
04-08-2015 15:53:57.673 -0700 INFO WatchedFile - Will begin reading at offset=16460 for file='/opt/splunk/splunkforwarder/var/log/splunk/splunkd-utility.log'.
04-08-2015 15:53:57.700 -0700 INFO WatchedFile - Will begin reading at offset=62969 for file='/opt/splunk/splunkforwarder/var/log/splunk/audit.log'.
04-08-2015 15:53:57.703 -0700 INFO WatchedFile - Will begin reading at offset=1317 for file='/opt/splunk/splunkforwarder/var/log/splunk/splunkd_stderr.log'.
04-08-2015 15:53:57.775 -0700 INFO WatchedFile - Will begin reading at offset=19758958 for file='/opt/splunk/splunkforwarder/var/log/splunk/metrics.log'.
04-08-2015 15:54:33.398 -0700 WARN DC:PhonehomeThread - No response to handshake for too long; starting over.
04-08-2015 15:54:33.398 -0700 WARN DC:PhonehomeThread - No response to handshake for too long; starting over.
04-08-2015 15:54:33.445 -0700 INFO DC:HandshakeReplyHandler - Handshake done.
04-08-2015 15:55:33.491 -0700 INFO DC:HandshakeReplyHandler - Handshake done.
Hi @Bobbs24
Please be sure that when responding to someone's answer, click on "Add comment" directly below their answer or, if responding to someone's comment, type in the "Add your comment..." box directly below their comment. You typed your response in the "Enter your answer here..." box at the very bottom of the page which, instead, posts a brand new answer. This will help with a clean continuous flow of the conversation and other users will know who you're responding to. Also, if the person who answered your question isn't following the question, then they won't get a notification that you posted something.
Your "answer" can no longer be converted to a comment since it is beyond the character limit. If you have a long response and are hitting a character limit, just break it up into multiple comments. Something to keep in mind from here on out. Thanks.
This means that you are referencing a group name in your outputs.conf that does not exist.
for example:
defaultGroup = heavy_forwarder
In order for that to be a valid configuration, you must have the appropriately named stanza like this:
[tcpout:heavy_forwarder]
server = splunkLB.example.com:4433
autoLB = true
[tcpout]
connectionTimeout = 45
defaultGroup = all_indexers
indexAndForward = 0
forwardedindex.0.whitelist = .*
[tcpout:all_indexers]
autoLB = true
server = heavyforwarder.hostname:9997
I believe this is correct as per your recommendation. Is it something I'm missing?