How do I reduce the number of Windows 4688 events generated by Splunk?

[joeldavideng]

While logging Windows 4688 events I noticed that the Splunkd process is actually responsible for generating over 90% of the events. I am currently dropping the events generated by the Splunkd process at a heavy forwarder, but I'd like to stop Splunkd from generating them in the first place since they take up disk space on my end points.

I believe the Splunkd process keeps launching child processes that check the Windows event logs and terminate. This would mean that the Splunk UF is spending most of its time monitoring itself. I would like to know if there is a way to reduce the number of child processes generated by Splunkd so that my endpoints generate fewer 4688 (Process Created) events.

[lmaclean]

Hi,

All of the windows inputs are disabled by default, on both the forwarders and the Windows TA unless you enabled them when installing the UF?

Also what are you trying to archive 1) Stop splunkd from creating sub-processes altogether; or 2) Reduce the amount of 4688 events being ingested?

For 1) yes the only way to stop it from generating new sub-processes is to stop it monitoring any events... Splunk along with applications like Chrome believe that creating sub processes for each task/tab is better than having one very large process i.e MS IE... So they have one main process that then spawns the sub-processes to monitor what you have configured.

If you want to just reduce the amount of events being ingested back into Splunk then you can use a blacklist filter... Also making sure that you restart splunkd on the forwarders to make sure the changes have taken effect.

From the default inputs.conf file:

... To set custom configurations, place an inputs.conf in $SPLUNK_HOME/etc/system/local/. For examples, see inputs.conf.example. You must restart Splunk to enable new configurations.

An Example:

[WinEventLog://Security] 
disabled = 0
blacklist.0 = EventCode="4688" Message=”.*New\sProcess\sName:\s+C:\\Program\sFiles\\Splunk\\bin\\splunk\-.*”

This will blacklist most of splunks sub-processes, there are a few like mongod, python, powershell that the above regex won't pick up but you can always just keep adding those as new blacklists (blacklist.1 etc...).

[joeldavideng]

Hi Imaclean,

I was attempting to achieve goal 1, to stop splunkd from creating the sub-processes in the first place. My issue was that setting the interval to 0 for the monitors doesn't actually do anything and I needed to set them to -1. Thanks for your thorough answer though.

Joel

[rayaivy]

Having the same issue. Black-Listing is easy... I'm trying to reach goal 1.

I have 4688's related to splunkd popping off 10 times in second, and it's eating through disk space. What files do I need to edit to stop splunk from creating so many processes???

[joeldavideng]

Rayaivy,

Create an app with an inputs.conf that defines the following

[admon]
disabled = 1
interval = -1

[MonitorNoHandle]
disabled = 1
interval = -1

[WinPrintMon]
disabled = 1
interval = -1

[perfmon]
disabled = 1
interval = -1

[powershell]
disabled = 1
interval = -1

[powershell2]
disabled = 1
interval = -1

[rayaivy]

What do each of these stanzas mean?

[joeldavideng · Accepted solution]

After digging further into this I found that all of the various Windows monitors are enabled by default and don't seem to respond to the disabled flag, but they will run once if you set the interval to -1.