Hello all.   I was reading over the article at https://www.splunk.com/en_us/blog/security/log4shell-detecting-log4j-vulnerability-cve-2021-44228-continued.html   Specifically at the New Outbound Traffic Detection with Baseline section.   Can someone explain to me the appendpipe's subsearch (I split it into parts but its actually one search) purpose and how it works?      | tstats summariesonly=false allow_old_summaries=true earliest(_time) as earliest latest(_time) as latest values(All_Traffic.action) as action values(All_Traffic.app) as app values(All_Traffic.dest_ip) as dest_ip values(All_Traffic.dest_port) as dest_port values(sourcetype) as sourcetype count from datamodel=Network_Traffic where (NOT (All_Traffic.dest_category="internal" OR All_Traffic.dest_ip=10.0.0.0/8 OR All_Traffic.dest_ip=172.16.0.0/12 OR All_Traffic.dest_ip=192.168.0.0/16 OR All_Traffic.dest_ip=100.64.0.0/10)) by All_Traffic.src_ip All_Traffic.dest_ip | rename "All_Traffic.*" as * | lookup egress_src_dest_tracker.csv dest_ip src_ip OUTPUT earliest AS previous_earliest latest AS previous_latest | eval earliest=min(earliest, previous_earliest), latest=max(latest, previous_latest) | fields - previous_* | appendpipe [ | fields src_ip dest_ip latest earliest | stats min(earliest) as earliest max(latest) as latest by src_ip, dest_ip | inputlookup append=t egress_src_dest_tracker.csv | stats min(earliest) as earliest max(latest) as latest by src_ip, dest_ip | outputlookup egress_src_dest_tracker.csv | where a=b ] | eventstats max(latest) as maxlatest | eval comparisonTime="-1h@h" | eval isOutlier=if(earliest >= relative_time(maxlatest, comparisonTime), 1, 0) | convert timeformat="%Y-%m-%dT%H:%M:%S" ctime(earliest),ctime(latest) ,ctime(maxlatest) | where isOutlier=1     I am trying to understand what this appendpipe portion is doing. Here is my current thought process: 0) It would take the result from the previous set of commands 1) summarize: latest/earlist by src/dest. 2) append the lookup 3) get the earliest/latest by src/dest again. (would the result be the same if we skipped #1?) 4) save the results 5) what does this where clause mean? There is no a or b  field that I can see.   Thanks!   ... View more

Hi all,   I configured an EDL and URL feed from Autofocus by following the steps in https://splunk.paloaltonetworks.com/autofocus-and-minemeld.html.  However, when I try to review the details from the macros from the link above,  no results are returned.   From the log file: /opt/splunk/var/log/splunk/Splunk_TA_paloalto_minemeld_feed.log I get the following entry for the EDL feed: 2021-01-05 15:29:16,550 ERROR pid=208666 tid=MainThread file=base_modinput.py:log_error:309 | Get error when collecting events. Traceback (most recent call last): File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/modinput_wrapper/base_modinput.py", line 128, in stream_events self.collect_events(ew) File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/minemeld_feed.py", line 72, in collect_events input_module.collect_events(self, ew) File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/input_module_minemeld_feed.py", line 84, in collect_events mmf_entries = get_feed_entries(helper, name, start, stats) File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/input_module_minemeld_feed.py", line 45, in inner ret_val = func(*args) File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/input_module_minemeld_feed.py", line 157, in get_feed_entries feed_entries = resp.json() File "/opt/splunk/etc/apps/Splunk_TA_paloalto/bin/splunk_ta_paloalto/aob_py3/requests/models.py", line 897, in json return complexjson.loads(self.text, **kwargs) File "/opt/splunk/lib/python3.7/json/__init__.py", line 348, in loads return _default_decoder.decode(s) File "/opt/splunk/lib/python3.7/json/decoder.py", line 340, in decode raise JSONDecodeError("Extra data", s, end) json.decoder.JSONDecodeError: Extra data: line 1 column 4 (char 3)   From the URL feed, I get: 2021-01-08 12:12:19,748 ERROR pid=15255 tid=MainThread file=base_modinput.py:log_error:309 | Failed to get entries for "af_daily": 401 Client Error: Unauthorized for url: https://autofocus.paloaltonetworks.com/output/threatFeedResult?v=json&tr=1   I have verified/retried the credentials and the API key (for Autofocus) to confirm that I have the correct value.     Note: I do get results from accessing the EDL/URL feeds manually via cURL.     Please let me know what else I can try. ... View more

Hello! I want to compare my event flow rate from the benchmark (last 21 - last 7 days [14 days in total] to the last 7 days to determine if there are any abnormal activities or to determine how my flow is trending. My process is to take the averages of the baseline and current along with the standard deviation to determine the zscore and work from that. The problem s that the search takes ~290 seconds and I'm hoping to see if there are any efficiencies that can be performed to make this work better. | tstats count where index=* earliest=-21d@d latest=-7d@d by _time index span=1h summariesonly=t | timechart span=1h sum(count) as count by index useother=f | fillnull value=0 | stats avg(*) as * | transpose 0 | rename "row 1" as avg column as index | appendcols [| tstats count where index=* earliest=-21d@d latest=-7d@d by _time index span=1h summariesonly=t | timechart span=1h sum(count) as count by index useother=f | fillnull value=0 | stats stdev(*) as * | transpose 0 | rename "row 1" as stdev column as index ] | appendcols [| tstats count where index=* earliest=-7d@d latest=@h by _time index span=1h summariesonly=t | timechart span=1h sum(count) as count by index useother=f | fillnull value=0 | stats avg(*) as * | transpose 0 | rename "row 1" as current column as index ] | eval z = (avg - current)/stdev Initially, instead of appendcols I used join but it seems appendcols is slightly faster. Part of the issue is that I can't get the stdev and avg in a single table easily (examples of my issues below). example: | tstats count where index=* earliest=-1d@d latest=@d by _time index span=1h summariesonly=t | timechart span=1h sum(count) as count by index useother=f _time index1 index2 index3 index4 index5 index6 index7 index8 index9 inedex10 2019-09-29 00:00 114929 109862 5447236 598915 101984 93383 1134374 3218677 135260 271187 2019-09-29 01:00 113735 94834 8043144 500234 101288 93374 1683179 3212936 268802 277495 2019-09-29 02:00 114069 99818 9472714 460066 99908 93260 1632044 3241514 857129 268001 ... If I use the following search, I can't seem to get the stdev and avg sorted by the index. Using stats avg(*) as avg_* stdev(*) by stdev_* by index gets me no results... And the following doesn't help with getting the avg/stdev by the index respectively so I had to resort to using the transpose 0 lines and joining the different searches together | tstats count where index=* earliest=-1d@d latest=@d by _time index span=1h summariesonly=t | timechart span=1h sum(count) as count by index useother=f | fillnull value=0 | stats avg(*) as avg_* stdev(*) as stdev_* avg_index1 avg_index2 avg_index3 ... stdev_index1 stdev_index2 stdev_index3 ... 79202.82857857 72468.4 7022379.3 ... 20705.25571 74106.285571 63430.8 ... Please let me know what else I can try to pretty this up. Thanks in advance! ... View more