How can you regulate resource consumption when monitoring a large number of files?

[Ron_Naken]

If a LWF has a large number of files to monitor, what settings can be used to help ensure that consuming/monitoring the files doesn't exceed 5% CPU utilization?

[gkanapathy]

I discovered today that if you do not use an explicit sourcetype assignment (either in inputs.conf or in a source stanza in props.conf) on the forwarder, then with a large number of files you will see a lot of CPU consumption. This is because the FileClassifier (i.e., the auto-sourcetyper) is very expensive to run.

Setting a default sourcetype in inputs.conf for each input stanza (and overriding it as needed in source stanzas in props.conf) will pretty much keep CPU usage down to around 2% of a core with most modern CPUs, assuming the default LWF thruput throttling of 256 kilobytes per second (which btw is over 20 GB/day on average).

Turning down the maxKBps if you don't have explicit sourcetypes assigned on a large (let's say a few hundred or a handful of thousand or more) will not be effective. You will have to throttle down to nearly-useless levels, and it still won't give satisfactory CPU consumption.

On the other hand, settings whitelists and blacklists so there is only a very small number of files being read (say, fewer than 20) will probably be effectively, but I'll assume that this is pretty fixed and you don't have the discretion to simply not index data.

[dwaddle · Accepted solution]

I don't think there is a way to directly influence / limit CPU utilization. Your most direct knob to turn on an LWF is its network bandwidth limit setting (maxKbps) in limits.conf. On the forwarder, place/edit ${SPLUNK_HOME}/etc/system/local/limits.conf with this stanza:

[thruput]
maxKBps = <integer>

It's not easy to guess exactly how low you will have to set that in order to get your CPU utilization requirement met.

I would caution you, though -- anything you do in this area to try to limit the work being done by a forwarder is usually counterproductive. You will be increasing the latency between when an event is logged and when it gets forwarded to your splunk index. In the end, this reduces Splunk's usefulness to you. You may reach a point where the forwarder can never 'catch up'.

You don't really say if the large number of files has a large frequency of change. If most of the files you are monitoring aren't changing then you probably want to adjust how your log directories / monitor statements are formed. If all of the files do have a large frequency of change it may be necessary to throw more CPU capacity at the problem in order to be able to keep everything monitored with reasonable latency.