My ouputs conf looks like this:
[tcpout]
defaultgroup = logstash
disabled = false
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.blacklist = (_audit|_internal|_introspection)
[tcpout:logstash]
server=localhost:7777
sendCookedData = false
useACK = true
As well as my actual events, I'm seeing loads of messages being emitted like this:
INFO Metrics - group=thruput, name=uncooked_output, instantaneous_kbps=0.176377, instantaneous_eps=0.096773, average_kbps=0.355449, total_k_processed=44.000000, kb=5.467773, ev=3.000000
INFO Metrics - group=thruput, name=thruput, instantaneous_kbps=0.176377, instantaneous_eps=0.096773, average_kbps=0.371606, total_k_processed=46.000000, kb=5.467773, ev=3.000000, load_average=0.030000
INFO Metrics - group=thruput, name=cooked_output, instantaneous_kbps=0.000000, instantaneous_eps=0.000000, average_kbps=0.000000, total_k_processed=0.000000, kb=0.000000, ev=0.000000
INFO Metrics - group=tcpout_connections, name=logstash:127.0.0.1:7777:0, sourcePort=8090, destIp=127.0.0.1, destPort=7777, _tcp_Bps=186.73, _tcp_KBps=0.18, _tcp_avg_thruput=0.39, _tcp_Kprocessed=46, _tcp_eps=0.10, kb=5.47
How can I eliminate these from the forwarder output?
New answer: what if you want to send some information to Splunk, but not everything?
Maybe you don't want the metrics, but you would like the errors, etc. from the splunkd.log
In $SPLUNK/HOME/etc/system/local/inputs.conf
, only disable the metrics log
[monitor://$SPLUNK_HOME/var/log/splunk/metrics.log]
disabled = true
You can set the "log levels" on the forwarder by copying $SPLUNK_HOME/etc/log.cfg
to $SPLUNK_HOME/etc/log-local.cfg
Edit $SPLUNK_HOME/etc/log-local.cfg
to customize the logging, but remember that these logs are a primary source for Splunk Monitoring Console. These edits will mostly affect the splunkd.log
There are many log channels, and you don't need to reset all of them. Just change "INFO" to "WARN" on any categories where you want to reduce the messages. You can delete any lines that you want to leave at INFO level. The following channels should always be left at INFO level:
category.TailingProcessor=INFO
category.loader=INFO
Thanks Iguinn, I know about the default directory, and I'll definitely try the log levels.
I migrated the whole indexer to a new Cloud instance, so there is no longer an issue with the tsids...but i'll test it out anyway.
Appreciate it
Splunk automatically forwards its internal logs. The inputs.conf
settings can be disabled to stop this. The settings may be found in several places, but usually they are set in $SPLUNK_HOMEetc/apps/SplunkUniversalForwarder/default/inputs.conf
Since you shouldn't edit anything in a default directory, create a local directory and create an inputs.conf
that contains
[monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log]
disabled = true
[monitor://$SPLUNK_HOME/var/log/splunk/metrics.log]
disabled = true
Do the same for $SPLUNK_HOME/etc/system/local/inputs.conf
[monitor://$SPLUNK_HOME/var/log/splunk]
disabled=true
If the problem continues, or you see other files in the tcp output stream, check all the inputs.conf files on your system. There may be a few other default inputs that you need to disable.
Hi there,
I know this is old and all, but is it still valid on version 7.0.1?
Adding the file:
$SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/local/inputs.conf
With content:
[monitor://$SPLUNK_HOME/var/log/splunk/metrics.log]
#_TCP_ROUTING = *
#index = _internal
disabled = true
Doesn't disable the metrics input.
(Trying to disable it since splunk-optimize goes crazy when trying to run on _internal index and ends up crashing the server out of memory).
Editing the default/inputs.conf also doesn't
Never edit the files in the default directories. Even if it works, your changes will be overwritten when you update Splunk. The files in the corresponding local directories always override the default directories.
This should still work in Splunk 7, but you are in the wrong directory. Do the same thing, but put it in $SPLUNK_HOME/etc/system/local
(which is probably /opt/splunkforwarder/etc/system/local
on a Linux box).
[monitor://$SPLUNK_HOME/var/log/splunk]
blacklist = metrics\.log