Splunk Search

Slow Response DNS Lookup Thousands of IPs

albyva
Communicator

I am running a DNS lookup on IP addresses using the following arrangement, but it is running very, very, very, slow because the initial input is thousands of IPs.

index=generic
| eval minute=strftime(_time, "%H:%M")
| stats sum(bytes) as totalbytes by dest_ip, minute
| eval Mbps=round(totalbytes/1024/1024)
| rename dest_ip as clientip
| lookup dnslookup clientip
| rename clienthost as destination
| table minute, Mbps, clientip, destination
| sort -Mbps
| head 10

Is there anyway this could be rewritten to speed up the Lookup process so that only the Top 10 address are run through Lookup and not the thousands I initially query?

My goal is just to really run the DNS lookup on the Top 10 addresses. But when I move
(head 10) in front of Lookup or Table, it doesn't give me the Top 10 from the Table/Sort.
Maybe this is a Splunk hierarchy problem or maybe I need a whole rewrite.
Any help is appreciated.

Tags (5)
0 Karma
1 Solution

Ayn
Legend

No, it's because you're moving head before the sort statement. Before you've sorted the events, Splunk will just grab the 10 first entries in whatever order they happen to come in (reverse chronologically). If you do this

index=generic | eval minute=strftime(_time, "%H:%M") | stats sum(bytes) as totalbytes by dest_ip, minute | eval Mbps=round(totalbytes/1024/1024) | sort -Mbps | head 10 | rename dest_ip as clientip | lookup dnslookup clientip | rename clienthost as destination | table minute, Mbps, clientip, destination

You should get the results you want.

View solution in original post

lakromani
Builder

this part

| rename dest_ip as clientip
| lookup dnslookup clientip
| rename clienthost as destination

could be done in one command

| lookup dnslookup clientip as dest_ip output clienthost as destination
0 Karma

Ayn
Legend

No, it's because you're moving head before the sort statement. Before you've sorted the events, Splunk will just grab the 10 first entries in whatever order they happen to come in (reverse chronologically). If you do this

index=generic | eval minute=strftime(_time, "%H:%M") | stats sum(bytes) as totalbytes by dest_ip, minute | eval Mbps=round(totalbytes/1024/1024) | sort -Mbps | head 10 | rename dest_ip as clientip | lookup dnslookup clientip | rename clienthost as destination | table minute, Mbps, clientip, destination

You should get the results you want.

albyva
Communicator

w00t. Worked like a charm. Thanks a million

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...