Universal Forwarder wineventlog handler affinity for domain PDC and not nearest DC when evt_dc_name not specifically defined

[dstaulcu]

When looking at TCP connections from domain-based hosts having universal forwarders (UF), we recently noticed that all UFs in our domain were maintaining connections to the instance of a domain controller (DC) having primary domain controller (PDC) emulator role and not to any of the nearer DCs that domain-based clients could more optimally distribute their load among. The count of TCP connections maintained to the PDC appears to be proportional to the count of event log channels we monitor via inputs.conf. As a result of this behavior, we discovered that we are exhausting resources on the PDC and contributing to user logon and application performance issues.

Our assumption all this time was that having evt_dc_name undefined would result in bind to nearest DC. In fact, it appears that the resultant behavior evt_dc_name being undefined changes from nearest DC to PDC somewhere between version UF v.6.0 and UF v6.03.

Our first reaction to this situation was to eliminate the need for communication with the PDC (in the short run) by setting evt_resolve_ad_obj = 0 for each event log monitor so that SID to CN translation is not performed and in turn PDC/DC connections should avoided. Changing evt_resolve_ad_obj to 0 does not seem to reduce the count of connections maintained. (Bug? - Certainly not efficient)

It appears that we can steer clients away from DC having PDC emulator role by specifying the DNS name of the domain in evt_dc_name. I'm not confident this will provide for most the efficient possible distribution of effort across available DCs.

The objective of this posting is:

(1) Heads up!
(2) What strategies are folks using to manage evt_resolve_ad_obj load distribution among DCs.
(3) Can anyone think of a good reason why the Splunk Agent would need to interface with PDC?

You can see evidence of dependency on PDC and port exhaustion of PDC with search of _internal index, splunkd log, having "Failed to retrieve the PDC in closest site."

[dstaulcu · Accepted solution]

I found a bizarre combination of input specifications that get us most of what we want.

It turns out splunk-winevtlog.exe will avoid creation of TCP connection to DC with evt_resolve_ad_obj = 0 ONLY if evt_dns_name is defined.

Interestingly, if you define evt_dns_name with DNS name of domain, splunk-winevtlog.exe will still use the PDC only.

If you define evt_dc_name with DNS name of domain, splunk-winevtlog.exe will bind to randomized DCs in the domain.

So, the winning combination is:

[WinEventLog://(System|Application|any other channels you don't want resolution for)]
evt_resolve_ad_obj = 0
evt_dc_name = dnsNameOfAdDomain
evt_dns_name = dnsNameOfAdDomain

[WinEventLog://Security]
evt_resolve_ad_obj = 1
evt_dc_name = dnsNameOfAdDomain
evt_dns_name = dnsNameOfAdDomain

This enables us to spread the load across available DCs and avoids unneeded TCP connections to DCs. This does not result in most efficient bandwidth by using nearest DC.

Ideally, specifying evt_dc_name as null would result in use of DsGetDCName function with flags not requiring return of PDC name but instead with default flags which would return nearest DC.

Note:
These behaviors and undocumented workarounds are true for Splunk Universal Forwarder v 6.0.3. The workarounds have not been tested on other versions of the Splunk Universal Forwarder.

[mendesjo]

Fascinating, any observations to this with newer versions of the clienet say 6.1x or better?

[dstaulcu]

Problem appears to have been corrected in Splunk 6.3

[gavsdavs_GR]

By "problem appears to have been corrected", do you mean the UF correctly looks for the DC in its local site ?
We're having to maintain an eventlogs app per AD site to manage distributed SID resolution.
Fixed in 6.3 ?

[dstaulcu]

Yes. We are seeing desired behavior now. UF are now interacting with in site DCs.

Also, IIRC, additional flags have been added to inputs.conf.spec providing additional control over DC communication.

[gavsdavs_GR]

awesome, thank you.