Monitoring Splunk

How to search splunkd.log to find tailing status of a file?

philip_wong
Communicator

How can we search splunkd.log effectively to know tailing status of a file and when is the last read?

0 Karma

philip_wong
Communicator

I think this is not the right answer. Almost nothing I can see after some noises filtered.

This is more close.

index=_internal component=WatchedFile

0 Karma

jnicholsenernoc
Path Finder

If you are using a heavy forwarder or indexing locally, you can query the _thefishbucket index to get run time information on the tailing of logs: “index=_thefishbucket"

Unfortunately, this isn't available on the univ/light forwarder.

It won't give you the exact timestamp of the last event exactly, but you can see the seek pointer and the last modified date and time. Based on this, you can see what it is doing (waiting for more log lines to be written), or processing a backlog, or stuck due to indexing being blocked and so forth.

More information:
http://blogs.splunk.com/2008/08/14/what-is-this-fishbucket-thing/

0 Karma

yannK
Splunk Employee
Splunk Employee

No this information is not logged in splunkd.log
Usually only the event of the first discovery of a file is listed, or when a batch processor finished to read a large file.
If you want the status of a file, use the REST-endpoint on the forwarder :
https://localhost:8089/services/admin/inputstatus/TailingProcessor:FileStatus

or use the file monitoring view in the SOS app on the local instance (if it has an UI)

Also you will never get the latest timestamp from the internal logs
For that you do a real time search on the indexed events, sort by _indextime and look at the most recent _time per source/host.

0 Karma

philip_wong
Communicator

Yes, I just realized SoS's "File Monitor Inputs" page is doing that. I think it's prefect for if using indexer to read logs. But badly, it doesn't work with forwarders. I'm not sure yet.

Behind the scenes an customized command bundled with SoS doing good job to parse TailingProcessor REST response

| tpstatusquery targetserver= namespace="system" owner="nobody" admin/inputstatus/TailingProcessor:FileStatus | table *

0 Karma

somesoni2
SplunkTrust
SplunkTrust

You can use metadata command as well for getting the most recent event timestamp by host/source/sourcetype

philip_wong
Communicator

This is a good start but not enough.
I wish to list all proceeded/processing logs by indexer/forwarder and the best also have the latest tailing timestamp

0 Karma

somesoni2
SplunkTrust
SplunkTrust

For starters, below can give all TailingProcessor activities including errors.

index=_internal component=TailingProcessor

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...