Monitoring Splunk

Why are _internal logs from heavy forwarder(HF) not getting to indexers after a Splunkd restart but _audit are?

Rob2520
Communicator

All of a sudden, _internal logs from HF stopped coming to indexers after a Splunkd restart. But, i see _audit logs making it to indexers. And, I see splunkd.log on HF is growing. There is no change in inputs.conf or outputs.conf before restart. What could be the reason?

0 Karma

adobrzeniecki
Path Finder

Run  /opt/splunk/bin/splunk btool outputs list --debug


You should see that the whitelisted index list does not include _internal. It is a precedence issue.  For us the issue was because the SplunkForwarder app did not include _internal in the whitelist for indexes. Just put this in /opt/splunk/etc/system/local/outputs.conf OR /opt/splunk/etc/SplunkForwarder/local/outputs.conf

[tcpout] 
forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry)

 

 

0 Karma

tiagofbmm
Influencer

Use btool to check on your inputs for splunkd.log files:

/opt/splunk/bin/splunk btool inputs list --debug | grep -B 5 log/splunk

If there is no TCP_ROUTING sending those to somewhere strange, check the /opt/splunk/var/log on the HF to check the modtime of splunkd.

More, do a tail -f on splunkd.log to check if these are being written

Finally, on your Search Head do a | tstats count where host=yourhf by index, _time

and check if something else has stopped meanwhile from that host

0 Karma

tiagofbmm
Influencer

@Rob2520 please accept an answer if it solved/helped it and upvote it. Otherwise let us know how can we help further

0 Karma

MuS
SplunkTrust
SplunkTrust

Check props.conf and/or transforms.conf if there is any filtering or routing configured. I know that _audit is not effected by those settings and therefore reaches your indexer. Also these kind of configuration changes need a Splunk restart to take effect.

cheers, MuS

0 Karma

Rob2520
Communicator

MuS, i don't see props or transforms related to splunkd logs.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...