Splunk Search

Unable to filter out lookup table values

browncardigan
Path Finder

I'm trying to filter out false-positive domains in a search of DNS events by using NOT on the ut_domain field of the lookup table. The search runs but provides no results, despite there being events that should've been returned in the results.

sourcetype=infoblox:dns record_type!=SRV record_type!=PTR record_type!=SOA
| eval list="mozilla"
| `ut_parse(query, list)`
| search ut_domain NOT [|inputlookup FP_entropy_domains.csv | fields ut_domain]
| `ut_shannon(ut_subdomain)`
| search ut_shannon >= 4.0
| stats count by ut_netloc ut_shannon
| sort -ut_shannon, -count

The search works fine until I try to filter out the domains.

I've tried multiple iterations including re-naming the lookup file's column name from ut_domain to domain and removing the ut_shannon >= 4.0 component, but nothing's worked:

| search NOT [|inputlookup FP_entropy_domains.csv | fields ut_domain]
| search ut_domain NOT [|inputlookup FP_entropy_domains.csv | fields domain]
| search NOT [|inputlookup FP_entropy_domains.csv | fields domain]  

Splunk indicates that it's successfully looking up the csv, and the comparison to the domain/ut_domain column in the lookup table is happening - I just don't get any results at the end of it.

Every answer I find on Splunk Answers indicates it should work with the simple |search NOT [|inputlookup | fields ] and I can't see what I'm doing wrong in this case.

0 Karma

to4kawa
Ultra Champion

Hi,

[|inputlookup FP_entropy_domains.csv | fields ut_domain]

returns (ut_domain=xx OR ut_domain=yy OR ....)

sourcetype=infoblox:dns  

What fields are extracted from this result?
If it is domain, you need to change the field name.

[|inputlookup FP_entropy_domains.csv | fields ut_domain|rename ut_domain as domain]
0 Karma

browncardigan
Path Finder
 sourcetype=infoblox:dns record_type!=SRV record_type!=PTR record_type!=SOA
 | eval list="mozilla"
 | `ut_parse(query, list)`
 | search ut_domain NOT [|inputlookup FP_entropy_domains.csv | fields ut_domain]
 | `ut_shannon(ut_subdomain)`
 | search ut_shannon >= 4.0
 | stats count by ut_netloc ut_shannon
 | sort -ut_shannon, -count

Line 1 pulls all non SRV, PTR, and SOA DNS records from the infoblox:dns sourcetype
Lines 2 - 3 use the URLToolbox App to parse the DNS queries using a list of Mozilla-recognised domains. This parses the domains out in the field "ut_domain".

I've tried putting the false-positive domains in the csv under both ut_domain and domain, and neither have worked. Using the rename command in the sub search wouldn't achieve anything different as far as I'm aware.

0 Karma

to4kawa
Ultra Champion

I think that the command is different when I see the following blog.
UT_parsing Domains Like House Slytherin
Random Words on Entropy and DNS

ut_parseut_parse_extended

| search  NOT [|inputlookup FP_entropy_domains.csv | fields ut_domain]
0 Karma

browncardigan
Path Finder

ut_parse and ut_parse_extended are the same thing according to the utbox documentation:

ut_parse(url, list) or ut_parse_extended(url, list):
Extended version of ut_parse_simple() which uses a list to extract the following
fields: ut_port, ut_domain, ut_tld, ut_domain_without_tld, ut_subdomain,
ut_subdomain_count and ut_subdomain_parts.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...