I'm running version 6.0.2 on CentOS 6. My DNS servers are a pair of Windows Server 2008 domain controllers. Every month, when I patch and reboot these Windows servers - which I do sequentially, Splunk writes logs to the database where host=[ip address] instead of host=[fqdn]. This breaks my alerting because my alerts are (mostly) defined by hostnames, for examplle:
host="DC*" AND "EventCode=4740"
While the logs are being written with host=[ip address], these alerts will never trigger.
Do I need to change the order of my dns servers listed in /etc/resolv.conf prior to rebooting my DNS servers? Or should I expect splunk to seamlessly send queries to the 2nd DNS to get a response?
Does splunk perform it's own DNS queries or does it rely on the underlying OS?
If it performs its own queries, is that configurable? Will changing the order of entries in /etc/resolv.conf require restarting Splunk?
I'd prefer to fix this wholly within Splunk, and without having to restart it monthly, because it takes 15 minutes to shut down.
