dnslookup very slow, odd results.

[oliverj]

(Splunk 7.2.3)
I have a single windows domain.
Inside that domain I have 2 subnets, 192.168.1.x, 192.168.2.x.
I have 19 hosts, spread across the 2 subnets.
All devices report their "host=" as an IP address, not a hostname
I ran the search:

index=x | dedup host | lookup clientip AS host OUTPUT clienthost AS hostname | table host forwarder hostname

And I have some issues with the results:
Issue 1)
When I inspect the search job, the "command.lookup" portion takes 217 seconds. This is searching the entire index of ~200 logs across the past week. The search with no lookup takes about 3 seconds to display the results.
I cannot find any logs relating to this delay, at least not in the /var/log/splunk directory. No timeouts or anything.

Issue 2)
Of my 19 hosts in the result table, only 9 actually have a "hostname" field. Closer inspection reveals that the missing hosts are all on the same subnet, the 192.168.2.x. For some reason, only one subnet is being pulled.
I have 2 search heads, one at each location, so I ran the same search from the other search head (192.168.2.x) . OPPOSITE RESULTS. All the 192.168.1.x hosts are missing the lookup data.

When I run a basic nslookup command from a workstation, the results and response time are identical for either subnet. So, I assume the script is doing something else, but I am not entirely sure which script is running this nslookup.

Is my search taking long because of timeouts for the failing subnet?
I changed my search to specify only one subnets worth of devices:

index=x subnet=1 | dedup host | lookup clientip AS host OUTPUT clienthost AS hostname | table host forwarder hostname

This result returns all 9 subnet1 host IPs with corresponding hostnames, but the search still took 71 seconds. So timeouts may have had a small part to play, but are definitely not the full culprit.

[jkat54 · Accepted solution]

Log into your search head and use nslookup to see if you can resolve the IPs in the problem subnet...

  nslookup 192.168.x.x

If nslookup fails then you have a DNS issue, not a splunk issue.

As for how long it takes the lookup to report back results... consider that the lookup is really just doing nslookup in the background from the server you execute the search on. Since the splunk lookup is single threaded, it will take time to run DNS queries for all the hosts. Especially if half the hosts are timing out the DNS queries which is what I suspect is your problem.

[oliverj]

This may be the issue -- my hostname->IP was working, but it seems I have no reverse DNS going.
Not sure why even 1 subnet works, because nslookup doesn't even pick up that even though splunk does, but for now I need to look into ptr records and reverse lookup zones.

I will revisit this thread and update -- hopefully this will resolve it.

[oliverj]

Revisiting:
My splunk admin spoke to my windows admin who spoke to DNS admin, and me myself and I realized that I had never enabled reverse lookup zones or PTR records.
Simple task to add that feature to the windows DNS server, and now everything reacts within a couple seconds.

Thank you to all for your help.

[jkat54]

Nice work!

[harsmarvania57]

Hi,

Can you please try below query, as far as I know dedup command require data on search head for processing.

index=x | stats count by host, forwarder | lookup dnslookup clientip AS host OUTPUT clienthost AS hostname | table host forwarder hostname

[oliverj]

Results look the same. From searchhead1, subnet1 has looked-up results but not subnet2.
Search took 170 seconds, vs the 217 on my original search.