Getting Data In

Is there a limit on the number of files a forwarder can monitor?

glitchcowboy
Path Finder

Is there a limit to the number of files an Universal Forwarder can monitor? When should I install multiple Universal Forwarder instances on a host?

We have several servers that will have about 200 log files to watch and each will be represented by a single [monitor] stanza in inputs.conf

Is there a performance bonus or problem with multiple forwarder instances?

1 Solution

gkanapathy
Splunk Employee
Splunk Employee

There is no hard limit, but there are practical ones depending on how many files are listed, how many of those are blacklisted, how many of those remaining are actively being written to, and how often. At the high end there is also question of how much latency in event indexing you are willing to tolerate. It also depends on how quickly your disk/filesystem can respond.

However, in the case of explicitly listing 200 files individually, that should be well within the ability of a forwarder to monitor with a second's delay. I would not worry unless you have on the order of 100,000 files listed and thousands being written to.

View solution in original post

glitchcowboy
Path Finder

Thanks (both of you) for your experience. I also had some feedback from support saying basically the same thing - there's no need for multiple forwarder instances and a single forwarder should do everything we need.

0 Karma

cwacha
Path Finder

We have one univeral forwarder running on a solaris system and it is monitoring around 30000 files where 1% of them is being constantly written to. Its working using the 4.3.3 code but we get delays of several minutes until events show up in the search GUI. If we only monitor one single file on this box the transport delay lies within seconds.

gkanapathy
Splunk Employee
Splunk Employee

There is no hard limit, but there are practical ones depending on how many files are listed, how many of those are blacklisted, how many of those remaining are actively being written to, and how often. At the high end there is also question of how much latency in event indexing you are willing to tolerate. It also depends on how quickly your disk/filesystem can respond.

However, in the case of explicitly listing 200 files individually, that should be well within the ability of a forwarder to monitor with a second's delay. I would not worry unless you have on the order of 100,000 files listed and thousands being written to.

Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...