Hi,
One of my Linux Forwarder not sending data to indexer. Could you please assist me what is wrong in my configuration.
Forwarder Conf:
inputs.conf
~~~~~~~~~
[default]
index = _Notifier
host = hostname.com
[monitor:///path/log/*]
disabled = false
followTail = 0
sourcetype = _Alerts
[monitor:///path/log/*]
disabled = false
followTail = 0
sourcetype = _Alerts
Outputs.conf:
~~~~~~~~~~~
[tcpout]
autoLB = true
compressed = true
defaultGroup = groupname
useACK=true
[tcpout:groupname]
server = splunk.hostname.com:9997
server.conf
[general]
serverName = xxxxx
[sslConfig]
sslKeysfilePassword = xxxxxx
[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder
[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free
On indexer
inputs.conf
~~~~~~~~~
[default]
host = xxxxxx
[splunktcp://9997]
disabled = 0
server.conf
~~~~~~~~~
[general]
serverName = xxxxxxx
[sslConfig]
sslKeysfilePassword = xxxxxxx
[lmpool:auto_generated_pool_download-trial]
description = auto_generated_pool_download-trial
quota = MAX
slaves = *
stack_id = download-trial
[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder
[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free
Few things can you verify it from UF to Indexer server.
Check the port connections as well firewall rules most of the time TCP error/Cooked data refer to these kind of issues.
Telnet to port and IP. etc
Try to troubleshoot problem from Network layer to Splunk you will find it.
I think this use of underscores is probably not a good idea. Index called _Notifier and sourcetype called _Alerts. If you don't want roles having access to the index, you can simply set up the roles that way. As for sourcetype with underscore the goal is not clear, so I can't comment.
This may actually be the cause of the data not being forwarded. There are some funny rules about _indexes being forwarded expressed in etc/system/default/outputs.conf because underscore indexes are considered to be internal to Splunk, and we want to avoid unnecessarily flooding user indexers with data that is not useful.
Incidentally,
Did you ever get an answer on resolving this as I am seeing the same thing on one of my Linux forwarders?
Hi,
I have changed index now, but getting error as below, could you please suggest me what can be done to fix this issue
On forwarders splunkd.log 10-13-2014 05:33:39.728 -0700 INFO TcpOutputProc - Connected to idx=x.x.x.x:9997 using ACK. 10-13-2014 05:33:39.732 -0700 WARN TimeoutHeap - Detected system time adjusted backwards by 1ms. 10-13-2014 05:33:54.765 -0700 WARN TimeoutHeap - Detected system time adjusted backwards by 1ms. 10-13-2014 05:33:58.057 -0700 WARN TimeoutHeap - Detected system time adjusted backwards by 1ms. 10-13-2014 05:34:09.746 -0700 WARN TcpOutputProc - Cooked connection to ip=x.x.x.x:9997 timed out on indexer metric.log getting connectionType=cooked.
the command ouput /opt/splunkforwarder/bin/splunk list forward-server Active forwards: indexername:9997 Configured but inactive forwards: None
Thanks!