Splunk Enterprise Security

Appendpipe or Multisearch for continuing processing after Outputlookup?

stroud_bc
Path Finder

I am attempting to create a custom Risk Attribution rule based on Web Proxy traffic to newly-seen (not-seen-before-yesterday) domains based off the blog post found below:

https://www.splunk.com/blog/2018/01/17/finding-new-evil-detecting-new-domains-with-splunk.html

What I am running into is the following: I want to use the "new domain monitoring" technique described in the above link, but I want to attribute risk to a USER. The problem is that I want to run my statistics WITHOUT the user field in order to update previously_seen_domains.csv and WITH the user field in order to create my risk attribution.

| tstats count from datamodel=Web where nodename=Web.Proxy by Web.url, Web.user, _time 
| rename Web.* AS * 
| eval list="mozilla" 
| `ut_parse_extended(url, list)` 
| appendpipe 
    [| stats earliest(_time) as earliest latest(_time) as latest by ut_domain
    | inputlookup append=t previously_seen_domains.csv 
    | stats min(earliest) as earliest max(latest) as latest by ut_domain
    | outputlookup previously_seen_domains.csv
    |search 1!=1
     ]
| stats earliest(_time) as earliest latest(_time) as latest by ut_domain, user 
| inputlookup append=t previously_seen_domains.csv 
| stats min(earliest) as earliest max(latest) as latest by ut_domain, user 
| eval isOutlier=if(earliest >= relative_time(now(), "-1d@d"), 1, 0) 
| convert ctime(earliest) ctime(latest) 
| where isOutlier=1

This search DOES work, but I don't know much about the internals of the |appendpipe command, and it seems like |multisearch might be more performant? Right now previously_seen_domains.csv has about 10k entries in it. If there are any suggestions on how to run the stats calculations once instead of twice with different aggregation, those would be helpful as well.

0 Karma
1 Solution

stroud_bc
Path Finder

I think I have a better understanding of |multisearch after reading through some answers on the topic. Additionally, for any future readers who are trying a similar approach, I found that the above search fails to respect the earliest values from the lookup, since the second | stats earliest(_time) as earliest latest(_time) as latest by ut_domain, user line ends up recalculating earliest values FOR EACH USER, which means the values will always fall within the relevant window and therefore will always end up attributing risk... To solve this, I turned to a much simpler and more linear approach using a subsearch with a separate |tstats command AFTER the outputlookup to join the user data to the newly-seen domain data.

View solution in original post

0 Karma

stroud_bc
Path Finder

I think I have a better understanding of |multisearch after reading through some answers on the topic. Additionally, for any future readers who are trying a similar approach, I found that the above search fails to respect the earliest values from the lookup, since the second | stats earliest(_time) as earliest latest(_time) as latest by ut_domain, user line ends up recalculating earliest values FOR EACH USER, which means the values will always fall within the relevant window and therefore will always end up attributing risk... To solve this, I turned to a much simpler and more linear approach using a subsearch with a separate |tstats command AFTER the outputlookup to join the user data to the newly-seen domain data.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...