Splunk Search

Looking to see if a string is present in another search

wweiland
Contributor

I'm looking through DNS logs in one index. They are normal DNS logs, so they have the normal query containing the host+domain. I have threat data which just has the domain in another index.

index=main sourcetype=dns 
| map search="index=sec u_type=domain u_status=Active earliest=0 latest=now u_indicator=$domain$"

above is what i've come up with so far, but it doesn't seem to work. Am I looking at this wrong? I want to make sure that if the domain in the threat data exist at all in the dns domain data, then I get a alert.

Any suggestions?

0 Karma
1 Solution

martin_mueller
SplunkTrust
SplunkTrust

If the number of threat domains is manageable, you can do something like this:

index=main sourcetype=dns [search index=sec u_type=domain u_status=Active earliest=0 latest=now | dedup u_indicator | fields u_indicator | rename u_indicator as domain]

If the number is larger, consider moving it to a lookup.

View solution in original post

0 Karma

aljohnson_splun
Splunk Employee
Splunk Employee

When you are using map, it is better to have a deduplicated list of the values you want to map.

For example:

index=main sourcetype=dns
| dedup domain
| table domain
| map search="index=sec u_type=domain u_status=Active earliest=0 latest=now u_indicator=$domain$"

Another way you could do it would be using the set command:

| set intersect
[ index=main sourcetype=dns | fields - _* | fields domain]
[ index=sec u_type=domain u_status=Active | fields - _* | fields u_indicator]

Which I personally find a little clearer.


Please remember - subsearches have limits - please look at sideview's answer here for an example of using lookups.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

If the number of threat domains is manageable, you can do something like this:

index=main sourcetype=dns [search index=sec u_type=domain u_status=Active earliest=0 latest=now | dedup u_indicator | fields u_indicator | rename u_indicator as domain]

If the number is larger, consider moving it to a lookup.

0 Karma

wweiland
Contributor

Would this only trigger if the domain exactly matches the u_indicator?

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Yes, so would your map search.

0 Karma

wweiland
Contributor

Can you think of a way I can write it so that if u_indicator is in any part of domain it would trigger? I was trying to think of how to use the if(match) but I can't seem to come up with the way.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Yeah, but it might be very slow:

index=main sourcetype=dns [search index=sec u_type=domain u_status=Active earliest=0 latest=now | dedup u_indicator | fields u_indicator | rename u_indicator as domain | eval domain = "*".domain."*"]
0 Karma

wweiland
Contributor

It also occurred to me that I could output the threat data to outputcsv and then do the same query against inputcsv? Can I still do the eval if I do that?

0 Karma

mreynov_splunk
Splunk Employee
Splunk Employee

evals are interpreted before lookups, but if they are in separate queries, should be no problem.

0 Karma

wweiland
Contributor

So the query works as written, but I ran into the problem that you suspected I would have. Trying to compare 50k threat domains to each record with a wildcard is taking 8 mins for a 15 min period. The only way I can think to resolve the problem would be to use a custom search command and drop it to python with dictionaries. Problem is, I don't control the Splunk back-end and can't add custom commands easily. Any suggestions on what I could do within native Splunk? Would lookup tables be faster than inputcsv?

0 Karma

wweiland
Contributor

Thank you kind sir. That works like a charm.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

With the lookup-based approach, you can set the lookup to match using wildcards and include those asterisks in the lookup.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...