Getting Data In

Monitoring Forward activity

ruiaires
Path Finder

I have a Splunk server that receives data from 2 normal (not light) forwarders.

In the forwarders, I had to create a local index with the same name as the server and I'm using local file monitors.

Everything is OK and data is getting to the server as it should.

The problem is, I can't seem to monitor the activity on the forwarders: 1. there are no events in the _internal indexes 2. the default activity searches return no data 3. in the main server, I also cannot find _internal index data related to the forwarder's activity

A few days ago, one of the forwaders had splunkd using > 10% of CPU and I have no way of knowing or diagnosing the problem.

So, I should I monitor the forwarders' activity ?

Tags (1)
1 Solution

tedder
Communicator

Forwarders don't forward their _internal activity by default. Add the following to your inputs.conf:

[monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log]
_TCP_ROUTING = * 
index = _internal

Here's a conversation about it.

View solution in original post

0 Karma

Simeon
Splunk Employee
Splunk Employee

You can utilize the settings in the LightForwarder app that send the internal logs onto the indexer. However, this only changes where the logs are located. To do this, you must edit the inputs.conf on your forwarder to reflect the following:

[monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log]
_TCP_ROUTING = *
index = _internal

[monitor://$SPLUNK_HOME/var/log/splunk/metrics.log]
_TCP_ROUTING = *
index = _internal

10% utilization is not uncommon for a full forwarder if it is doing some heavy processing. Without more detail of your inputs, this will be hard to diagnose.

0 Karma

tedder
Communicator

Forwarders don't forward their _internal activity by default. Add the following to your inputs.conf:

[monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log]
_TCP_ROUTING = * 
index = _internal

Here's a conversation about it.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...