Getting Data In

What is/isn't getting monitored and why?

drawks
Explorer

Is there a way to see what files are being read by the various monitor/fschange stanzas in input.conf?

1 Solution

jrodman
Splunk Employee
Splunk Employee

There are a few ways.

The command line has an invocation 'splunk list monitor' which will show you files that splunk found that it thinks it's supposed to read. Generally it will list files that it was configured to read which have no new data, so this is more of a way to validate that the configuration agrees with your file layout than to see what's live.

You can enable more verbose logging to see what's going on. http://www.splunk.com/wiki/Community:Troubleshooting_Monitor_Inputs If you enable category.TailingProcessor to higher output levels via $SPLUNK_HOME/etc/log-local.cfg or via the manager screens, then you'll see greater detail in splunkd.log about what files are being looked at, included, excluded etc. You can search this with splunk, looking at index=_internal.

There's also a protoype endpoint (4.1+) available at https://your.instance:yourport/services/admin/inputstatus/TailingProcessor:FileStatus You can see things like eliminated for crc-collision reasons, eliminated for binary status, didn't match whitelist, matched blacklist, and so on. This can be used remotely on forwarders, so long as the default admin password has been changed (or you've allowed remote login anyway).

Note:

  • This is the splunkd management port, not the splunk web interface.
  • http*s*
  • It will complain about security in most browsers, because it's a self-signed certificate
  • This interface is likely to move or change in future releases, so automating against it is probably not a good idea

View solution in original post

jrodman
Splunk Employee
Splunk Employee

There are a few ways.

The command line has an invocation 'splunk list monitor' which will show you files that splunk found that it thinks it's supposed to read. Generally it will list files that it was configured to read which have no new data, so this is more of a way to validate that the configuration agrees with your file layout than to see what's live.

You can enable more verbose logging to see what's going on. http://www.splunk.com/wiki/Community:Troubleshooting_Monitor_Inputs If you enable category.TailingProcessor to higher output levels via $SPLUNK_HOME/etc/log-local.cfg or via the manager screens, then you'll see greater detail in splunkd.log about what files are being looked at, included, excluded etc. You can search this with splunk, looking at index=_internal.

There's also a protoype endpoint (4.1+) available at https://your.instance:yourport/services/admin/inputstatus/TailingProcessor:FileStatus You can see things like eliminated for crc-collision reasons, eliminated for binary status, didn't match whitelist, matched blacklist, and so on. This can be used remotely on forwarders, so long as the default admin password has been changed (or you've allowed remote login anyway).

Note:

  • This is the splunkd management port, not the splunk web interface.
  • http*s*
  • It will complain about security in most browsers, because it's a self-signed certificate
  • This interface is likely to move or change in future releases, so automating against it is probably not a good idea

drawks
Explorer

Sure! Point you browser to the splunkd host in question

https://localhost:8089/services/admin/inputstatus/TailingProcessor:FileStatus

to get a full break down of all the files that splunk is aware of, their status, and the reason for their status. I.E.

  • /var/log/apache/access.log
    parent /var/log
    type could not read

  • /var/log/acpid.1.gz
    parent /var/log
    type Did not match whitelist '(.log|log$|messages$|mesg$|cron$|acpid$|.out)'.

  • /opt/splunk/var/log/splunk/splunkd.log
    file position 642076
    file size 642076
    parent $SPLUNK_HOME/var/log/splunk
    percent 100.00
    type open file

Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...