Hello
my splunkd.log shows the the following error
ERROR TcpOutputProc - LightWeightForwarder/UniversalForwarder not configured. Please configure outputs.conf.
i am pushing my outputs using my deployment server . the directory structure looks like
/apps/splunkforwarder/etc/deployment-apps/outputs/local/outputs.conf
and my outputs.conf looks like
[indexer_discovery:env-masternode]
master_uri = https://masternode:8089
pass4SymmKey = XXXXXXXXX
[tcpout]
defaultGroup = primary_indexers
forceTimebasedAutoLB = true
maxQueueSize = 7MB
useACK = true
[tcpout:primary_indexers]
autoLB = true
indexerDiscovery = env-masternode
and i verified that the outputs is downloaded from deployment server to my universal forwarder an it is under
/apps/splunkforwarder/etc/apps/outputs/local/outputs.conf
all configs looks fine. but why am getting this error??
./splunk cmd btool outputs list
this is the command i used
You output is hard to read...
Here an example from my test env:
/etc/system/default/outputs.conf [syslog]
/etc/system/default/outputs.conf dropEventsOnQueueFull = -1
/etc/system/default/outputs.conf maxEventSize = 1024
/etc/system/default/outputs.conf priority = <13>
/etc/system/default/outputs.conf type = udp
/etc/apps/fwd_sendtoindexer/local/outputs.conf [tcpout]
/etc/system/default/outputs.conf ackTimeoutOnShutdown = 30
/etc/system/default/outputs.conf autoLBFrequency = 30
/etc/system/default/outputs.conf blockOnCloning = true
/etc/system/default/outputs.conf blockWarnThreshold = 100
/etc/system/default/outputs.conf compressed = false
/etc/system/default/outputs.conf connectionTimeout = 20
/etc/apps/fwd_sendtoindexer/local/outputs.conf defaultGroup = default-autolb-group
/etc/system/default/outputs.conf disabled = false
/etc/system/default/outputs.conf dropClonedEventsOnQueueFull = 5
/etc/system/default/outputs.conf dropEventsOnQueueFull = -1
/etc/system/default/outputs.conf forceTimebasedAutoLB = false
/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.0.whitelist = .*
/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.1.blacklist = _.*
/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.2.whitelist = (_audit|_introspection)
/etc/apps/SplunkUniversalForwarder/default/outputs.conf forwardedindex.filter.disable = false
/etc/system/default/outputs.conf heartbeatFrequency = 30
/etc/system/default/outputs.conf indexAndForward = false
/etc/system/default/outputs.conf maxConnectionsPerIndexer = 2
/etc/system/default/outputs.conf maxFailuresPerInterval = 2
/etc/system/default/outputs.conf maxQueueSize = auto
/etc/system/default/outputs.conf readTimeout = 300
/etc/system/default/outputs.conf secsInFailureInterval = 1
/etc/system/default/outputs.conf sendCookedData = true
/etc/system/default/outputs.conf sslQuietShutdown = false
/etc/system/default/outputs.conf tcpSendBufSz = 0
/etc/system/default/outputs.conf useACK = false
/etc/system/default/outputs.conf writeTimeout = 300
/etc/apps/fwd_sendtoindexer/local/outputs.conf [tcpout-server://10.204.240.180:9997]
/etc/apps/fwd_sendtoindexer/local/outputs.conf [tcpout:default-autolb-group]
/etc/apps/fwd_sendtoindexer/local/outputs.conf server = 10.204.240.180:9997
splunk list forward-server
Active forwards:
None
Configured but inactive forwards:
10.204.240.180:9997
I would switch of idxAck and IndexerDiscovery until you have everything up and running...
Are you sure your input is active?
Maybe keep it simple?
My outputs.conf...
[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = 10.204.240.180:9997
[tcpout-server://10.204.240.180:9997]
Hi,
please run
/apps/splunkforwarder/bin/splunk btool outputs list --debug | less
on your forwarder and check whether the outputs.conf is really active.
Did the forwarder do a restart?
HTH,
Holger
if really i have problem with my outputs, i even should not see my internal logs. but all my internal logs are being forwarded and i can query them.
Please send the output of btool and search for TcpOutput error messages in index=_internal.
is your forwarder included in the search results if you do the following search?
| tstats count where index=* OR index=_* by host,index
HTH,
Holger
OUTPUT OF BTOOL:
[indexer_discovery:env-masternode]
master_uri = https://masternode:8089
pass4SymmKey = XXXXX
[syslog]
dropEventsOnQueueFull = -1
maxEventSize = 1024
priority = <13>
type = udp
[tcpout]
ackTimeoutOnShutdown = 30
autoLBFrequency = 30
blockOnCloning = true
blockWarnThreshold = 100
compressed = false
connectionTimeout = 20
defaultGroup = primary_indexers
disabled = false
dropClonedEventsOnQueueFull = 5
dropEventsOnQueueFull = -1
forceTimebasedAutoLB = true
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_introspection|_internal)
forwardedindex.filter.disable = false
heartbeatFrequency = 30
indexAndForward = false
maxConnectionsPerIndexer = 2
maxFailuresPerInterval = 2
maxQueueSize = 7MB
readTimeout = 300
secsInFailureInterval = 1
sendCookedData = true
sslQuietShutdown = false
tcpSendBufSz = 0
useACK = true
writeTimeout = 300
[tcpout:primary_indexers]
autoLB = true
indexerDiscovery = env-masternode
No results found for TcpOutput error messages in index=_internal
output of | tstats count where index= OR index=_ by host,index
sqa01-ins01-scc51-dbs01 _internal 873298
there are no errors in outputs.
and the output of my ./splunk list forward-server
Active forwards:
None
Configured but inactive forwards:
None