Getting Data In

How do I configure Universal Forwarder to not send INFO Metrics over TCP?

markdixon
Explorer

My ouputs conf looks like this:

[tcpout]
defaultgroup = logstash
disabled = false

forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.blacklist = (_audit|_internal|_introspection)

[tcpout:logstash]
server=localhost:7777
sendCookedData = false
useACK = true

As well as my actual events, I'm seeing loads of messages being emitted like this:

 INFO  Metrics - group=thruput, name=uncooked_output, instantaneous_kbps=0.176377, instantaneous_eps=0.096773, average_kbps=0.355449, total_k_processed=44.000000, kb=5.467773, ev=3.000000
 INFO  Metrics - group=thruput, name=thruput, instantaneous_kbps=0.176377, instantaneous_eps=0.096773, average_kbps=0.371606, total_k_processed=46.000000, kb=5.467773, ev=3.000000, load_average=0.030000
 INFO  Metrics - group=thruput, name=cooked_output, instantaneous_kbps=0.000000, instantaneous_eps=0.000000, average_kbps=0.000000, total_k_processed=0.000000, kb=0.000000, ev=0.000000
 INFO  Metrics - group=tcpout_connections, name=logstash:127.0.0.1:7777:0, sourcePort=8090, destIp=127.0.0.1, destPort=7777, _tcp_Bps=186.73, _tcp_KBps=0.18, _tcp_avg_thruput=0.39, _tcp_Kprocessed=46, _tcp_eps=0.10, kb=5.47

How can I eliminate these from the forwarder output?

lguinn2
Legend

New answer: what if you want to send some information to Splunk, but not everything?
Maybe you don't want the metrics, but you would like the errors, etc. from the splunkd.log

In $SPLUNK/HOME/etc/system/local/inputs.conf, only disable the metrics log

 [monitor://$SPLUNK_HOME/var/log/splunk/metrics.log]
 disabled = true

You can set the "log levels" on the forwarder by copying $SPLUNK_HOME/etc/log.cfg to $SPLUNK_HOME/etc/log-local.cfg
Edit $SPLUNK_HOME/etc/log-local.cfg to customize the logging, but remember that these logs are a primary source for Splunk Monitoring Console. These edits will mostly affect the splunkd.log
There are many log channels, and you don't need to reset all of them. Just change "INFO" to "WARN" on any categories where you want to reduce the messages. You can delete any lines that you want to leave at INFO level. The following channels should always be left at INFO level:

category.TailingProcessor=INFO
category.loader=INFO

alexsayegh
Explorer

Thanks Iguinn, I know about the default directory, and I'll definitely try the log levels.
I migrated the whole indexer to a new Cloud instance, so there is no longer an issue with the tsids...but i'll test it out anyway.
Appreciate it

0 Karma

lguinn2
Legend

Splunk automatically forwards its internal logs. The inputs.conf settings can be disabled to stop this. The settings may be found in several places, but usually they are set in $SPLUNK_HOMEetc/apps/SplunkUniversalForwarder/default/inputs.conf

Since you shouldn't edit anything in a default directory, create a local directory and create an inputs.conf that contains

[monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log]
disabled = true

[monitor://$SPLUNK_HOME/var/log/splunk/metrics.log]
disabled = true

Do the same for $SPLUNK_HOME/etc/system/local/inputs.conf

[monitor://$SPLUNK_HOME/var/log/splunk]
disabled=true

If the problem continues, or you see other files in the tcp output stream, check all the inputs.conf files on your system. There may be a few other default inputs that you need to disable.

alexsayegh
Explorer

Hi there,
I know this is old and all, but is it still valid on version 7.0.1?
Adding the file:

$SPLUNK_HOME/etc/apps/SplunkUniversalForwarder/local/inputs.conf

With content:

[monitor://$SPLUNK_HOME/var/log/splunk/metrics.log]
#_TCP_ROUTING = *
#index = _internal
disabled = true

Doesn't disable the metrics input.
(Trying to disable it since splunk-optimize goes crazy when trying to run on _internal index and ends up crashing the server out of memory).

0 Karma

alexsayegh
Explorer

Editing the default/inputs.conf also doesn't

0 Karma

lguinn2
Legend

Never edit the files in the default directories. Even if it works, your changes will be overwritten when you update Splunk. The files in the corresponding local directories always override the default directories.

This should still work in Splunk 7, but you are in the wrong directory. Do the same thing, but put it in $SPLUNK_HOME/etc/system/local (which is probably /opt/splunkforwarder/etc/system/local on a Linux box).

0 Karma

law
Observer

[monitor://$SPLUNK_HOME/var/log/splunk]
blacklist = metrics\.log

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...