Running Splunk 4.2.3 on CentOS 5.3 x64 to capture syslog data sourced from network devices. I needed to enable DNS resolution in a way that makes it easy to search events based on domain name. I ran into problems until I eventually got it working, and wanted to share my findings.
I noticed that DNS resolution is unpredictable when done via the UDP port (connection_host = dns). Watching tcpdump I saw successful PTR queries occurring, but events for only a few devices started showing the host field as the FQDN in search results. One device in particular this would consistently work on, while the others it didn't. This device was sending by far the most syslog traffic. The other two were also some of the noisier boxes. When I did a lookup and used a new field as the output the hostname was there; Splunk had the data but wouldn't do a rewrite of the host field via the UDP connection consistently. It seemed to slowly work through them, as gradually more and more network device events would show domain names vs. IP.
Regarding some other posts about what and where to put configuration info to enable DNS lookup, in the file /opt/splunk/etc/system/default/transforms.conf there is already a transform for the command external_lookup.py - except the syntax is incorrect. It uses clienthost and clientip instead of host and ip, respectively. I noticed it says example there, but it can be confusing because it still shows up under Lookups. Seems easiest just to modify the entry there to work.
Lastly, I still could not get rewrite to work to make the host field show the host name vs IP until I created an alias for the host field (host AS hostip) and used the alias for the lookup.
* | lookup dnslookup ip AS hostip OUTPUT host as host
The above worked to rewrite the host field with the results of the rDNS lookup.
