Universal Forward, Windows and Event Log files why is it so slow?

[colinj]

Howdy all,

We are running in to a problem with the speed of a universal forwarder on one of our Windows servers (2008 R2 64bit).

Every two hours the Windows server will contact each of the eight domain controllers, get back all of the successful and failed login events for the past two hours and outputs those events to a saved event log file (.evtx). One file is created for each of the domain controllers for each two hour block. So over the course of the day we produce 12 files for each domain controller for a total of 96 files.

The forwarder on the windows server is watching the directory that the files will appear in and then forwarder on the contents of the files to out indexers. The universal forwarder is not keeping up with the amount of data being generated which is about ~700 MB for each two hour period. So what I'm wondering is what might be cause the lag? The performance is slow enough that the data is being generated faster than it can be forwarded.

I've turned up the maxKBps to 1024 in the limits.conf file for the forwarder but that does not seem to have helped. Can anyone suggest what else we might look at?

Please and thank you

Colin J.

[gkanapathy]

I would really not recommend polling events remotely from domain controllers. I'm also not so familiar with the evtx monitoring, but it would not surprise me if quite simply it is bottlenecking on that in two ways. First, because it's only handling one file at a time, and second, because the parsing of the evtx file is too slow. It seems likely to me that the Splunk Windows evtx parsing wasn't specifically designed for high throughput. The expected use, especially under this load, is to collect the data directly from the machines via API.

[colinj]

The Windows Admins are preventing me. They don't like installing "agents" on their domain controllers.

[iunderwood]

What is preventing you from installing universal forwarders on your domain controllers and using those instead?