Getting Data In

How to configure inputs.conf on a universal forwarder to ignore monitoring and indexing folders that are older than 1 day?

vaibhavagg2006
Communicator

Hi

I am monitoring a folder which has high level of nesting and daily, 1000's of folders gets created. The name of the folder is unique based on some id. I am seeing a delay of 10-12 hours in getting the logs which are placed deep in the nth folder. I believe this is because Splunk checks for each and every folder sequentially for a match. Can we ignore folders older than 1 day so that Splunk does not search inside old folders? I am using a universal forwarder with good bunch of indexers to index the data. There is no throughput issue. The daily ingestion is around 1-2 gigs.
Below is my inputs.conf stanza

[monitor:///<folder path>]
_TCP_ROUTING = prod
ignoreOlderThan = 2d
whitelist = .log
index = index1
sourcetype = sample_sourcetype
disabled = 0

Please provide your inputs on this issue.

0 Karma

ddrillic
Ultra Champion

http://docs.splunk.com/Documentation/Splunk/6.3.3/Data/Monitorfilesanddirectorieswithinputs.conf covers it.
ignoreOlderThan = 2d seems to be the right set-up.

0 Karma

vaibhavagg2006
Communicator

I believe "ignoreOlderThan" will only ignore files. My problem is splunk is taking too much time in traversing through the folders to find a match.

0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...