All of a sudden, _internal logs from HF stopped coming to indexers after a Splunkd restart. But, i see _audit logs making it to indexers. And, I see splunkd.log on HF is growing. There is no change in inputs.conf or outputs.conf before restart. What could be the reason?
Run /opt/splunk/bin/splunk btool outputs list --debug
You should see that the whitelisted index list does not include _internal. It is a precedence issue. For us the issue was because the SplunkForwarder app did not include _internal in the whitelist for indexes. Just put this in /opt/splunk/etc/system/local/outputs.conf OR /opt/splunk/etc/SplunkForwarder/local/outputs.conf
[tcpout] forwardedindex.2.whitelist = (_audit|_internal|_introspection|_telemetry)
Use btool to check on your inputs for splunkd.log files:
/opt/splunk/bin/splunk btool inputs list --debug | grep -B 5 log/splunk
If there is no TCP_ROUTING sending those to somewhere strange, check the /opt/splunk/var/log on the HF to check the modtime of splunkd.
More, do a tail -f on splunkd.log to check if these are being written
Finally, on your Search Head do a | tstats count where host=yourhf by index, _time
and check if something else has stopped meanwhile from that host
@Rob2520 please accept an answer if it solved/helped it and upvote it. Otherwise let us know how can we help further
Check props.conf
and/or transforms.conf
if there is any filtering or routing configured. I know that _audit
is not effected by those settings and therefore reaches your indexer. Also these kind of configuration changes need a Splunk restart to take effect.
cheers, MuS
MuS, i don't see props or transforms related to splunkd logs.