We are currently running an evaluation of Splunk. Our current environment exists out of one indexer and 105 Windows servers that have Splunk Forwarder installed. All Forwarders use the same configuration:
monitor one directory, with a whitelist option, non-recursive
one location in the Windows Event log.
All servers (that have a Forwarder installed) run Windows Server 2008R2. On most systems, the Forwarder uses little resources, while on others, there are constant spikes of 100% CPU.
It seems that only small servers are impacted by this. By small I mean one (virtual) CPU and not a lot of system activity. I have used Procmon to analyse what's going on and to compare the splunkd.exe process on a busy system (where it runs fine) and on a small system (where it uses lot's of CPU).
During the CPU spikes there are a lot of QueryDirectory actions seen on the systems that have these issue. The directory is the one that's in the monitor stanza. The action happens +- 150000 on troubled systems compared to +-300 for systems that run fine (roughly same monitor period.)
The configuration is the same, the forwarders were all installed the same way, using the command line. What could cause the Forwarder to query that directory so much and cause so much CPU?
... View more