Greetings,
I'm looking to craft a correlation that allows me to compare the results between two separate searches. Here's the use case:
For instance, maybe something along the lines of the logic below:
index=Threat_Intelligence
| table DomainName
| where DomainName IN [search index=DNS | table RequestedDomain]
FYI: The latest Threat Intelligence feeds are pulled every single morning and is updated within Splunk. I thought about using lookup tables or KV Store lookups, but we're pulling in several files each morning, 2 of which are close to 1GB in size. It looks like Splunk Cloud caps the event limit of these lookups to 10,000 events by default, and I've read to be cautious about increasing this limit.
Try something like this:
index=Threat_Intelligence OR index=DNS
| eval DomainName=coalesce(DomainName,RequestedDomain)
| stats dc(index) as indexes by DomainName
| where indexes=2
(This will list DomainNames which appear on both indexes)
Try something like this:
index=Threat_Intelligence OR index=DNS
| eval DomainName=coalesce(DomainName,RequestedDomain)
| stats dc(index) as indexes by DomainName
| where indexes=2
(This will list DomainNames which appear on both indexes)
Thank you, this works like a charm. I see the logic, essentially just checking to see if a particular domain exists in both indexes (Threat_Intelligence and DNS), which would indicate a hit.
Reading this again, I think you're looking for a subsearch. Something along these lines.
index=DNS
[| search index=Threat_Intelligence
| table DomainName
| rename DomainName as RequestedDomainName
| format]
| dedup RequestedDomainName
| table RequestedDomainName
So the subsearch returns a string like ((RequestedDomainName=IntelDomainName1) OR (RequestedDomainName="IntellDomainName2" etc etc)". Run the subsearch on its own and you'll see what it builds. This string gets applied to your DNS data as a filter. Leaving all the events in the DNS index which have domain names referenced in the Threat_Intelligence index.
Thank you, this appears to work well also. It's seems to be a little less efficient, but still gets the job done.
Thanks @avajax0 . I do agree my solution, isn't as as elegant as the one posted by @somesoni2 , but I think mine might actually have performance advantages. Suggesting this here in case anyone would like to comment. The way I see it, in my example, the subquery returns a filter list which is applied to the DNS index. DNS might potentially have thousand of host names and hopefully the threat intelligence index will only have a hand full. Therefore the subquery in my example could reduce the number of events fetched from the DNS index by an order of magnitude improving the overall performance of the query. "Filter as early as possible" is a best practice rule I've heard repeated often. Interested to hear any other opinions. Good luck!
Some example events from each index would be useful.