In splunk 4.* and 5.* I used to have Splunk Universal forwarders on Windows Domain Controllers.
The volume of events on the channels : Security, Application, System etc... were high during the daytime, and the forwarder couldn't keep up. Over the day, we were experiencing delay between the events timestamp and the _indextime (once indexed and searchable). And during the night the events were catching up.
You could see the behavior with a search like this for the previous day :
sourcetype=WinEventLog host=myhost | delay_sec=_indextime-_time | timechart avg(delay_sec) by sourcetype
But since I switched to Universal forwarders on 6.0, things started to get better.
What magic happened ?
PS : To boost speed :
- We already removed the default thruput limit, and were having unlimited speed in limits.conf
[thruput]
maxKBps=0
- and the AD object name resolver was the local AD.
The magic is called Modular Inputs, introduced in Splunk 6.0
Most of the Windows inputs were rewritten to use them, and for the WinEventLog, they improved the performance.
see the modular inputs description http://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/ModInputsIntro
and the WinEventLog
see http://docs.splunk.com/Documentation/Splunk/latest/Data/MonitorWindowsdata
The old input version were hitting performance degradation above many events per seconds, and the delay was proportional to the number of events.
With modular inputs since 6.0, the performance improved because :
I recommend to upgrade to 6.0 to benefit of this improvement.
Also if you cannot upgrade all your infrastructure, 6.0 forwarders are compatible with 5.0 indexers.
The magic is called Modular Inputs, introduced in Splunk 6.0
Most of the Windows inputs were rewritten to use them, and for the WinEventLog, they improved the performance.
see the modular inputs description http://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/ModInputsIntro
and the WinEventLog
see http://docs.splunk.com/Documentation/Splunk/latest/Data/MonitorWindowsdata
The old input version were hitting performance degradation above many events per seconds, and the delay was proportional to the number of events.
With modular inputs since 6.0, the performance improved because :
I recommend to upgrade to 6.0 to benefit of this improvement.
Also if you cannot upgrade all your infrastructure, 6.0 forwarders are compatible with 5.0 indexers.