I'm trying to use "Monitor Files & Directories" as data input. I got two Data Input sources,
One is script that runs every 10 min and puts a data file on Splunk file system (/opt/splunk/var/ps_search/)
Second data input is the "Monitor Files & Directories" that basically is supposed to look under the /opt/splunk/var/ps_search directory and index all the incoming files.
The incoming files are of "csv" type and have unique file name (timestamp in the file name). I see only the first csv file getting indexed and not the subsequent ones that are generated by the script. I've read http://answers.splunk.com/questions/4103/directory-monitoring-not-picking-up-new-files and http://www.splunk.com/base/Documentation/latest/Admin/Monitorfilesanddirectories, but not sure what else I need to do. Few Questions,
In the documentation it says the monitor would only check for new files every 24 hours - is that right? How else can I make it to continously look for new files in the directly? Do I need to use crawl?
Is it possible to use monitor to do the above and when the file is indexed delete that file (similar to using sinkhole)?
In my case once a file is copied into the directory it's not changed, so I basically just want to delete it once Splunk has indexed it.
... View more