Getting Data In

Universal Forward, Windows and Event Log files why is it so slow?

colinj
Path Finder

Howdy all,

We are running in to a problem with the speed of a universal forwarder on one of our Windows servers (2008 R2 64bit).

Every two hours the Windows server will contact each of the eight domain controllers, get back all of the successful and failed login events for the past two hours and outputs those events to a saved event log file (.evtx). One file is created for each of the domain controllers for each two hour block. So over the course of the day we produce 12 files for each domain controller for a total of 96 files.

The forwarder on the windows server is watching the directory that the files will appear in and then forwarder on the contents of the files to out indexers. The universal forwarder is not keeping up with the amount of data being generated which is about ~700 MB for each two hour period. So what I'm wondering is what might be cause the lag? The performance is slow enough that the data is being generated faster than it can be forwarded.

I've turned up the maxKBps to 1024 in the limits.conf file for the forwarder but that does not seem to have helped. Can anyone suggest what else we might look at?

Please and thank you

Colin J.

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

I would really not recommend polling events remotely from domain controllers. I'm also not so familiar with the evtx monitoring, but it would not surprise me if quite simply it is bottlenecking on that in two ways. First, because it's only handling one file at a time, and second, because the parsing of the evtx file is too slow. It seems likely to me that the Splunk Windows evtx parsing wasn't specifically designed for high throughput. The expected use, especially under this load, is to collect the data directly from the machines via API.

colinj
Path Finder

The Windows Admins are preventing me. They don't like installing "agents" on their domain controllers.

0 Karma

iunderwood
Path Finder

What is preventing you from installing universal forwarders on your domain controllers and using those instead?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...