Limit Memory used by forwarder on Domain Controller

[wbfoxii]

All of our Domain Controllers are VMs with limited resources. We have the UF running on them, catching Windows Event Log Security, and admon. The forwarder uses 2.2 GB of memory, and the administrator is flipping out when he sees memory usage warnings from OpenView. Restarting causes memory usage to plunge, but it eventually grows back to this level. I realize that the forwarder is maintaining a cache in case it can't transfer the data up to the indexer. Is there any way to cap this? I have heard that Splunk holds the memory "loosely" and will release it if it is required, but I've got no evidence of that to support my argument.

[wbfoxii]

I got the bad news:

Hello Bill,

There isn't a way to limit the amount of memory being used by a forwarder directly. What you can do is to turn off inputs and/or descrease the threshold of inputs that run on an interval, like scripted inputs such as perfmon/wmi/etc.

However, Splunk recommends that we use 5.0.3 of the forwarder:
If you want to move to 5.0.3, that is fine too, it contains a number of fixes for splunk-admon.exe related to memory.

So I'm not happy about the memory use, but I'd rather have the log data.

[splunk68]

So you get the memory usage problem on both DCs over LAN and WAN, right ?
When you perform a search targeting one of this DC, do you see a big time gap between the time you run the search and the timestamp of the last search results ?
If yes, I'd say either the forwarder can't keep up with the events rate through the WAN, or the indexer can't keep up generally with the amount of incoming events.
One of those bottlenecks could explain the huge memory usage ?

[wbfoxii]

DCs are both WAN and LAN connected to the splunk indexer.

Events in a 24-hour period varies based on the site the DC provides service. So between 240,000 and 3 million.

[splunk68]

Are your DCs forwarding over LAN or WAN ? How many events are you indexing over a 24h time period / DC ?

[wbfoxii]

We tried useACK = true in the outputs.conf to see if that would make a difference in the memory used. Inconclusive.

[wbfoxii]

The forwarders vary from 4.3.2 to 5.0.1. They are all running on Windows Server 2008.

[bmacias84]

What version on which platform?

[wbfoxii]

I saw a note about that, but it seemed that would only reduce memory usage by a few MB. I need to reduce by a GB.

Inputs are WinEventLog:Security and the ad monitor.

Indexer acknowledgement is a new one on me!

I'm sure that everything is at a default value
maxQueueSize=500KB

[bmacias84]

Have you modified your queue sizes for your inputs and outputs confs? Is indexer acknowledgement turned on? What are all your inputs you are forwarding?