Getting Data In

How to configure inputs.conf on a universal forwarder to ignore monitoring and indexing folders that are older than 1 day?

vaibhavagg2006
Communicator

Hi

I am monitoring a folder which has high level of nesting and daily, 1000's of folders gets created. The name of the folder is unique based on some id. I am seeing a delay of 10-12 hours in getting the logs which are placed deep in the nth folder. I believe this is because Splunk checks for each and every folder sequentially for a match. Can we ignore folders older than 1 day so that Splunk does not search inside old folders? I am using a universal forwarder with good bunch of indexers to index the data. There is no throughput issue. The daily ingestion is around 1-2 gigs.
Below is my inputs.conf stanza

[monitor:///<folder path>]
_TCP_ROUTING = prod
ignoreOlderThan = 2d
whitelist = .log
index = index1
sourcetype = sample_sourcetype
disabled = 0

Please provide your inputs on this issue.

0 Karma

ddrillic
Ultra Champion

http://docs.splunk.com/Documentation/Splunk/6.3.3/Data/Monitorfilesanddirectorieswithinputs.conf covers it.
ignoreOlderThan = 2d seems to be the right set-up.

0 Karma

vaibhavagg2006
Communicator

I believe "ignoreOlderThan" will only ignore files. My problem is splunk is taking too much time in traversing through the folders to find a match.

0 Karma
Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Wednesday, May 29, 2024  |  11AM PST / 2PM ESTRegister now and join us to learn more about how you can ...

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer Certification at ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...