Getting Data In

Parallel data monitor/transmission - inputs.conf precedence

rgaleone1
Path Finder

Splunk:

Indexer <- series of tubes -> Forwarder

App:

fwdtosplunk/default/inputs.conf
[monitor:///path1/]
[monitor:///path2/]

Question:

Both path1 are path2 are large directories. Will the Forwarder need to completely send all data in path1 before beginning to sending data from path2?
0 Karma

kristian_kolb
Ultra Champion

The TailingProcessor, which is 'responsible' for checking which files have been updated (or are unread), used to be a single-threaded process in v5 (don't know if that has changed). Unless I'm much mistaken, that probably means that it would handle the input files sequentially.

However, if not all of those files are being updated (i.e. it's an archive of ooold files), you would only see this problem the first time you're indexing the files. Also, if it is an archive of old files that you DON'T want to be indexed, you could set the ignoreOlderThan parameter in inputs.conf or move the old files away to some other directory (don't put them in subdirectory, unless you also set recurse=false for that input. 🙂

Hope this helps a little,

K

Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...