Windows Event log volume is extremely high on one server - where is data being pulled from?

[cwilmoth]

I have seen a few other questions about huge data volumes surrounding Windows Event log processing, but I haven't seen a clear answer as to the cause.

We have a group of servers that have the System and Security event log monitors active. I am watching 10-15MB of license usage being consumed per minute with these servers. If I search based on indextime > xxx and look at the _time values, they are in fact older events. However, when I talk to the Windows admin and have him look at the System log on the server he reports a count of around 75,000. I have close to 30 million events and still counting for this one server (average of 500 events per second). Where can it be pulling this data from? It doesn't look like it is duplicate data.

We need to get a handle on this prior to deploying to the rest of our production Windows hosts. I appreciate any help in targeting the problem here.

Thanks.

[masonmorales]

Take a look at Splunk Utilization Monitor (SUM) on splunkbase: https://splunkbase.splunk.com/app/2678/

It has a dashboard that can help you isolate what is consuming your license.

[Runals]

Agree with dolivasoh - there is a good chance these are historical events coming in. Once that glut has processed what I would do as a next step is to see what the EventCodes (eventID) are and see if the majority are from a particular type of activity like the Windows firewall or someone having turned on object access auditing. If you have a 6.x forwarder you can chose to not bring in a particular event type at all or you could look at rewriting the logs if there is data of value. I wrote something up to that effect a while back (link).

[dolivasoh]

This certainly happens once you first turn them on. Windows hosts will always send quite a bit more than Linux. The key is to index only what you need. Take a look at routing and filtering data and remember, this is why we're allowed 5 violations in a month. Cram them all in on the same day or stagger them out every weekend and you'll be good.