Getting Data In

How to monitor a single file to be indexed by modifying inputs.conf?

chrisduimstra
Path Finder

I'm having trouble getting a single file to be indexed. I have successfully monitored all files in a directory before, but I'm not sure what is causing the specified file to not be monitored. Here are two stanzas I have tested in inputs.conf, neither with success.

[monitor://C:\Program Files (x86)\Sell\Sell.FT Service\Client.log]
index = ClientProcessor
sourcetype = Client_log
disabled = false

[monitor://C:\Program Files (x86)\Sell\Sell.FT Service]
index = ClientProcessor
sourcetype = Client_log
disabled = false
whitelist = (.*\.log)

EDIT:
I created a new .txt file and copied a couple logs over to the new file. I then added another stanza to monitor that file, and the new file was indexed but not the original. I have tried .txt and .log suffixes for LPClient. Here is the current inputs.conf

[monitor://C:\Program Files (x86)\Sell\Sell.FT Service\LPClient.txt]
index = LP
sourcetype = LPClient_log
disabled = 0

[monitor://C:\Program Files (x86)\Sell\Sell.FT Service\NewTextDocument.txt]
index = LP
sourcetype = LPClient_log
disabled = 0
0 Karma
1 Solution

MuS
SplunkTrust
SplunkTrust

here is a list of troubleshooting steps:

Check splunk btool inputs list monitor if your config is applied correctly and check splunk list inputstatus if Splunk is reading the directory and the files?
Check if your service account has permission to read files in that directory?
Check on any intermediate parsing layer, if there are nullQueues configured?
Check index=_internal sourcetype=splunkd source=*metrics.log series=*Sell.FT Service* over all time to get some information if data was sent?
Search the index over all time, maybe you have some timestamping issue?
Last resort tcpdump the traffic to see if the input instance is sending out events and they get lost somewhere?
Was the file already indexed, if so clean the fish bucket index so Splunk will re-index it or use the option crcSalt on your input - search docs for more information on both topics.

This list is almost never-ending ... Good luck and I hope you find the missing puzzle piece.

cheers, MuS

View solution in original post

MuS
SplunkTrust
SplunkTrust

here is a list of troubleshooting steps:

Check splunk btool inputs list monitor if your config is applied correctly and check splunk list inputstatus if Splunk is reading the directory and the files?
Check if your service account has permission to read files in that directory?
Check on any intermediate parsing layer, if there are nullQueues configured?
Check index=_internal sourcetype=splunkd source=*metrics.log series=*Sell.FT Service* over all time to get some information if data was sent?
Search the index over all time, maybe you have some timestamping issue?
Last resort tcpdump the traffic to see if the input instance is sending out events and they get lost somewhere?
Was the file already indexed, if so clean the fish bucket index so Splunk will re-index it or use the option crcSalt on your input - search docs for more information on both topics.

This list is almost never-ending ... Good luck and I hope you find the missing puzzle piece.

cheers, MuS

chrisduimstra
Path Finder

Thanks for the info. splunk list inputstatus led me to this, type = unreadable file type. I am unsure how to resolve this as the file type is .log and it is an auto-generated log file.

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi chrisduimstra,

just converted it to an answer, feel free to accept it if it answered your question.

cheers, MuS

0 Karma

chrisduimstra
Path Finder

I can't answer why, but I opened one of the files and saved it, and splunk then indexed the logs across all hosts.

0 Karma

deangoris
Explorer

I'm having the exact same thing. file type keeps reporting "unreadable file type" until I open te file and save it (without any changes made). After savind the file gets indexed and gets status "Finished reading".
Did you get this issue solved?

0 Karma

MuS
SplunkTrust
SplunkTrust

Do you get any warnings or errors in splunkd.log related to this files? As well to list the inputs using btool to see if your config is being applied:

 splunk btool inputs list

Also check the inputs status to see if Splunk is monitoring the directory:

splunk list inputstatus

Hope this helps ...

cheers, MuS

chrisduimstra
Path Finder

The last logs in splunkd.log are for the download and install of the app from the deployment server. The input stanza to monitor the directory is listed with the btool command. However, I am unable to run the last command due to privileges on my account.

Here's the entry from btool

[monitor://C:\Program Files (x86)\Sell\Sell.FT Service]
_rcvbuf = 1572864
allowSslCompression = true
allowSslRenegotiation = true
baseline = 0
dedicatedIoThreads = 2
disabled = false
enableSSL = 1
evt_dc_name = 
evt_dns_name = 
evt_resolve_ad_obj = 0
host = APPS01
index = ClientProcessor
interval = 60
maxSockets = 0
maxThreads = 0
port = 8088
sourcetype = Client_log
sslVersions = *,-ssl2
useDeploymentServer = 0
whitelist = (.*\.log)
0 Karma

MuS
SplunkTrust
SplunkTrust

Looking at the whitelist option .... why do you use a capturing group here? Have you tried to use

 whitelist = .*\.log
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...