On more than a few of my domain controllers, the Splunk Universal Forwarder is consuming 100% CPU and spewing many errors in splunkd.log like this:
06-22-2015 15:26:58.603 -0400 ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe (/splunk-winevtlog.exe)"" splunk-winevtlog - EvtDC::connectToDC: DsBind failed: (5)
This appears to be an issue with the SID resolution as I am collecting Windows Logs on these domain controllers. I'm aware of the evt_dc_name parameter in inputs.conf, but I don't wish to use it because the objects should all be available locally. How do I resolve this issue?
I was able to confirm that a security control (Symantec Critical Server Protection / DSP) was preventing the Splunk service from doing what it wanted to do. Resolving the security rules fixed the issue.
I was able to confirm that a security control (Symantec Critical Server Protection / DSP) was preventing the Splunk service from doing what it wanted to do. Resolving the security rules fixed the issue.
According to MSDN RPC error code 5 is ERROR_ACCESS_DENIED which definitely gives credence to @dolivasoh's theory of this being a problem that could easily land one in the 7th circle. Are you running the UF as a domain user account? There's also discussion about what user you should run Splunk as on Windows and what permissions said user should have at a base level in the docs.
UniversalForwarder+Windows-Permissions=HELL
Make sure you have adequate permissions to do all things specified on the forwarder. Not a complete solution but a good place to start.
Running Splunk 6.2.0 Forwarder in most cases, will upgrade to a newer revision if that is a confirmed fix.