Indexing latency arising at forwarders?

[sowings]

I've got a few Splunk universal forwarders, running 4.3.3 on a 64-bit Linux. These systems are monitoring approximately 700 different files, according to splunk list monitor. Only a few of these are of significant use, and the rest are mostly static.

The "important" log files are high volume, accumulating as much as 6G / day. I'm seeing a significant bit of latency in indexing the data from these files. If I run a search for this data, I sometimes observe as much as two hours of delay in getting the data into Splunk (that is, the "most recent" events from a given source (file) are two hours old). I can tail the file on disk and see recent events, so I believe that the latency is within the forwarder.

I haven't observed any error messages in splunkd.log, though I do occasionally see

INFO  WatchedFile - Will begin reading at offset=0 for file='/logs/hostfoo/logfile.log'.

The time stamps of these messages are scattered throughout the day, even though the logfile is only rotated once daily. It's as though Splunk is busy going through the list of static files, and taking a long time to get back to the "busy" logfiles to index the new events.

The ulimit for the number of open files has been raised from the default of 1024 to a high number, so I don't think it's related to file handles.

Any hints or triage steps appreciated.

[sowings · Accepted solution]

It turns out that the log volume on this particular host was overwhelming the default bandwidth limit of the Universal Forwarder, so the apparent lag was caused by a full pipeline. We upped the limit from the default of 256KB/s to 1024KB/s in limits.conf.

[dart]

One thing that might help is to set ignoreOlderThan in your inputs.conf to ignore the files that are static.

[monitor:///my/folder]
ignoreOlderThan=1d

[sowings]

We ended up removing the monitor statement for the inactive tree.

[dart]

It's done based on modtime, so as long as the modtime gets update it will get indexed.

[sowings]

I have lingering questions about ignoreOlderThan: first, does Splunk ever re-check a file that it has ignored in this way? That is, if that file does eventually change, would Splunk noticed?

Second, does Splunk still occasionally check this file to see if it's been updated?

Overall, I'm wondering whether this stanza monitoring static files will still constitute a "diversion" from indexing the main files I'm interested in.