Environment has one search head and one search peer. Data is sent to a directory [item (1)] configured to be monitored and indexed by the search peer. Both the search head and search peer have the same "indexes.conf" entry for the index [see item 21)], and the index is showing up in the search head GUI. Search peer has entry in "inputs.conf" to monitor the directory where data is being sent [see item (3)]. When a file is copied into the directory, the expected behavior is for the file to be ingested into Splunk and consequently be searchable; however this behavior is not occurring.
We have other indexes on this environment that do work as intended, but for some reason this particular setup is not working. Any and all help would be appreciated.
Item (1)*
/my/file/dir_ectory
Item (2)
[MY_in_dex]
homePath = $SPLUNK_DB/MY_in_dex
thawedPath = $SPLUNK_DB/thawedpath/MY_in_dex
coldPath = $SPLUNK_DB/coldpath/MY_in_dex
Item (3)
[monitor:///my/FILE/dir_ectory]
index = My_in_dex
*[NOTE: this traversal does start from "/" on a *nix machine]
Thanks to everyone for their help!
It ended up being a latency issue, where the data too up to an hour to be ingested.
Your indexes.conf
is highly unusual (but I don't see why it shouldn't work); why is it not like this:
homePath = $SPLUNK_DB/MY_in_dex/db
coldPath = $SPLUNK_DB/MY_in_dex/colddb
thawedPath = $SPLUNK_DB/MY_in_dex/thaweddb
What role do you have, you can check the indexes under your role to which you have access. Probably this is not listed.
HI,
did you check the splunkd log on forwarder and Indexer??
You should have logs from the forwarder which stats that its trying to read the file which should be monitored.
If not you should check if it picked up the inputs.conf at all with btool. Maybe you did not restart the forwarder after deploying the inputs.conf to it?
In which order did you create the input and the index? somethimes if you deploy the input before creating the index the forwarder tries to send logs and its not getting it written to the index, so in the fishbucket its saved that it read it but it wasn´t indexed.
Maybe a permission issue? Check what indexes you are allowed to search
Or your indexer is not excepting the data being sent by the forwarder? You could check the _internal for tcpin connections from the forwarder to the indexer
You can also elevated the log level:
On the Splunk instance that is monitoring the files, navigate to the $SPLUNK_HOME/etc directory and edit the file:
log.cfg
modify the following settings and change INFO to DEBUG
category.TailingProcessor=INFO
category.WatchedFile=INFO
category.ArchiveProcessor=INFO
category.TailReader=INFO
save the file.
Restart the Splunk instance.
Take a look at the log: $SPLUNK_HOME/var/log/splunk/splunkd.log
Look for the names of the files you were monitoring, the debug information should tell you why they were skipped.
Set the values back to INFO after you figure out your problem.
source: https://answers.splunk.com/answers/439546/how-to-troubleshoot-why-monitored-files-in-a-direc.html