I'm trying to filter out false-positive domains in a search of DNS events by using NOT on the ut_domain field of the lookup table. The search runs but provides no results, despite there being events that should've been returned in the results.
sourcetype=infoblox:dns record_type!=SRV record_type!=PTR record_type!=SOA
| eval list="mozilla"
| `ut_parse(query, list)`
| search ut_domain NOT [|inputlookup FP_entropy_domains.csv | fields ut_domain]
| `ut_shannon(ut_subdomain)`
| search ut_shannon >= 4.0
| stats count by ut_netloc ut_shannon
| sort -ut_shannon, -count
The search works fine until I try to filter out the domains.
I've tried multiple iterations including re-naming the lookup file's column name from ut_domain to domain and removing the ut_shannon >= 4.0 component, but nothing's worked:
| search NOT [|inputlookup FP_entropy_domains.csv | fields ut_domain]
| search ut_domain NOT [|inputlookup FP_entropy_domains.csv | fields domain]
| search NOT [|inputlookup FP_entropy_domains.csv | fields domain]
Splunk indicates that it's successfully looking up the csv, and the comparison to the domain/ut_domain column in the lookup table is happening - I just don't get any results at the end of it.
Every answer I find on Splunk Answers indicates it should work with the simple |search NOT [|inputlookup | fields ] and I can't see what I'm doing wrong in this case.
Hi,
[|inputlookup FP_entropy_domains.csv | fields ut_domain]
(ut_domain=xx OR ut_domain=yy OR ....)
sourcetype=infoblox:dns
What fields are extracted from this result?
If it is domain
, you need to change the field name.
[|inputlookup FP_entropy_domains.csv | fields ut_domain|rename ut_domain as domain]
sourcetype=infoblox:dns record_type!=SRV record_type!=PTR record_type!=SOA
| eval list="mozilla"
| `ut_parse(query, list)`
| search ut_domain NOT [|inputlookup FP_entropy_domains.csv | fields ut_domain]
| `ut_shannon(ut_subdomain)`
| search ut_shannon >= 4.0
| stats count by ut_netloc ut_shannon
| sort -ut_shannon, -count
Line 1 pulls all non SRV, PTR, and SOA DNS records from the infoblox:dns sourcetype
Lines 2 - 3 use the URLToolbox App to parse the DNS queries using a list of Mozilla-recognised domains. This parses the domains out in the field "ut_domain".
I've tried putting the false-positive domains in the csv under both ut_domain and domain, and neither have worked. Using the rename command in the sub search wouldn't achieve anything different as far as I'm aware.
I think that the command is different when I see the following blog.
UT_parsing Domains Like House Slytherin
Random Words on Entropy and DNS
ut_parse
→ ut_parse_extended
| search NOT [|inputlookup FP_entropy_domains.csv | fields ut_domain]
ut_parse and ut_parse_extended are the same thing according to the utbox documentation:
ut_parse(url, list) or ut_parse_extended(url, list):
Extended version of ut_parse_simple() which uses a list to extract the following
fields: ut_port, ut_domain, ut_tld, ut_domain_without_tld, ut_subdomain,
ut_subdomain_count and ut_subdomain_parts.