Splunk Search

Search using join returning incoherent results

lightech1
Path Finder

Hello,

I am getting different results running the same search over the same interval of time.
The search is:

sourcetype=incapsula (siteid="*" OR sourceServiceName="*") Action=REQ_BLOCKED* CEF_Severity>=0 src=* | transaction src | join src  [search sourcetype=cisco:asa message_id=302013 OR message_id=302015 OR message_id=302020  Cisco_ASA_action=allowed src=* | transaction src] | table src, CEF_Name, Action, request, Cisco_ASA_action, dest_ip, dest_port

As I am running a subsearch, I suspect that may be a timeout problem as I know that by default the time limit for subsearches is 60 seconds and this subsearch is taking around 69 seconds to run.
If this is the problem, can you help me making the search more performant?

Another strange thing is that when I "inspect" de job, I can see that the normalized search includes some conditions that I didn't include in the search.

This is a sample of the "Normalized search" on the job inspector:

Normalized search:

litsearch ( sourcetype="cisco:asa" OR sourcetype=cisco_asa ) message_id=302013 OR message_id=302015 OR message_id=302020 ( ( ( ( sourcetype="cisco:asa" OR sourcetype=cisco_asa ) ) AND ( ( ( Cisco_ASA_vendor_action=built ) ) OR ( ( Cisco_ASA_vendor_action=permitted ) ) OR ( ( Cisco_ASA_vendor_action="Pre-allocated" ) ) OR ( ( Cisco_ASA_vendor_action=whitelisted ) ) OR ( ( Cisco_ASA_vendor_action=Rebuilt ) ) OR ( ( Cisco_ASA_vendor_action="permitted tcp" ) ) OR ( ( Cisco_ASA_vendor_action="permitted udp" ) ) OR ( ( Cisco_ASA_vendor_action="permitted icmp" ) ) OR ( ( Cisco_ASA_vendor_action="access requested" ) ) OR ( ( Cisco_ASA_vendor_action="access permitted" ) ) OR ( ( Cisco_ASA_vendor_action=accessed ) ) ) ) OR ( ( ( sourcetype="cisco:fwsm" OR sourcetype=cisco_fwsm ) ) AND ( ( ( Cisco_ASA_vendor_action=built ) ) OR ( ( Cisco_ASA_vendor_action=permitted ) ) OR ( ( Cisco_ASA_vendor_action="Pre-allocated" ) ) OR ( ( Cisco_ASA_vendor_action=whitelisted ) ) OR ( ( Cisco_ASA_vendor_action=Rebuilt ) ) OR ( ( Cisco_ASA_vendor_action="permitted tcp" ) ) OR ( ( Cisco_ASA_vendor_action="permitted udp" ) ) OR ( ( Cisco_ASA_vendor_action="permitted icmp" ) ) OR ( ( Cisco_ASA_vendor_action="access requested" ) ) OR ( ( Cisco_ASA_vendor_action="access permitted" ) ) OR ( ( Cisco_ASA_vendor_action=accessed ) ) ) ) OR ( ( ( sourcetype="cisco:pix" OR sourcetype=cisco_pix ) ) AND ( ( ( Cisco_ASA_vendor_action=built ) ) OR ( ( Cisco_ASA_vendor_action=permitted ) ) OR ( ( Cisco_ASA_vendor_action="Pre-allocated" ) ) OR ( ( Cisco_ASA_vendor_action=whitelisted ) ) OR ( ( Cisco_ASA_vendor_action=Rebuilt ) ) OR ( ( Cisco_ASA_vendor_action="permitted tcp" ) ) OR ( ( Cisco_ASA_vendor_action="permitted udp" ) ) OR ( ( Cisco_ASA_vendor_action="permitted icmp" ) ) OR ( ( Cisco_ASA_vendor_action="access requested" ) ) OR ( ( Cisco_ASA_vendor_action="access permitted" ) ) OR ( ( Cisco_ASA_vendor_action=accessed ) ) ) ) ) OR ( Cisco_ASA_action=allowed ) ( ( ( ( sourcetype="bluecoat:proxysg:access:file" ) AND ( ( c_ip="" ) ) ) OR ( ( ( sourcetype="bluecoat:proxysg:access:syslog" OR sourcetype=bluecoat ) ) AND ( ( c_ip="" ) ) ) OR ( ( sourcetype="cisco:ios" ) AND ( ( src_ip="" ) ) ) OR ( ( sourcetype=cpu ) AND ( ( host="" ) ) ) OR ( ( sourcetype=df ) AND ( ( host="" ) ) ) OR ( ( sourcetype=iis ) AND ( ( c_ip="" ) ) ) OR ( ( sourcetype="mcafee:epo" ) AND ( ( src_ip="" ) ) ) OR ( ( sourcetype="mcafee:ids" ) AND ( ( SIP="" ) ) ) OR ( ( sourcetype="oracle:listener:text" ) AND ( ( CLIENTIP="" ) ) ) OR ( ( sourcetype="oracle:listener:xml" ) AND ( ( CLIENTIP="" ) ) ) OR ( ( sourcetype=ps ) AND ( ( host="" ) ) ) OR ( ( source="" ) AND ( ( host="" ) ) ) OR ( ( source="" ) AND ( ( host="" ) ) ) OR ( ( source="Perfmon" ) AND ( ( host="" ) ) ) OR ( ( source="WMI" ) AND ( ( host="" ) ) ) OR ( ( sourcetype=vmstat ) AND ( ( host="" ) ) ) ) OR ( ( src="" OR sourcetype=aix_secure OR sourcetype=linux_secure OR sourcetype=osx_secure OR sourcetype=syslog ) ) OR ( ( sourcetype="cisco:asa" OR sourcetype=cisco_asa ) ) OR ( ( sourcetype="cisco:fwsm" OR sourcetype=cisco_fwsm ) ) OR ( ( sourcetype="cisco:pix" OR sourcetype=cisco_pix ) ) OR ( sourcetype="oracle:audit:text" ) OR ( sourcetype="oracle:audit:xml" ) ) | litsearch sourcetype=cisco:asa message_id=302013 OR message_id=302015 OR message_id=302020 Cisco_ASA_action=allowed src= | fields keepcolorder=t "*" "_bkt" "_cd" "_si" "_txn_ends_with" "_txn_filter_match" "_txn_starts_with" "host" "index" "linecount" "source" "sourcetype" "splunk_server" "src" | pretransaction src

Why is this happening?

As this is a client's enviroment, I don't have access to the filesystem to check the .conf files.

Can you help me with this issue?

Thank you in advance.
Regards

0 Karma
1 Solution

somesoni2
SplunkTrust
SplunkTrust

Give this a try

Updated

(sourcetype=incapsula (siteid="*" OR sourceServiceName="*") Action=REQ_BLOCKED* CEF_Severity>=0 src=*) OR ( sourcetype=cisco:asa message_id=302013 OR message_id=302015 OR message_id=302020  Cisco_ASA_action=allowed src=* ) 
| fields sourcetype src, CEF_Name, Action, request, Cisco_ASA_action, dest_ip, dest_port
| stats dc(sourcetype) as sts values(CEF_Name) as CEF_Name values(Action) as Action values(request) as request values(Cisco_ASA_action) as Cisco_ASA_action values(dest_ip) as dest_ip values(dest_port) as dest_port by src 
| where sts=2 | fields - sts

View solution in original post

somesoni2
SplunkTrust
SplunkTrust

Give this a try

Updated

(sourcetype=incapsula (siteid="*" OR sourceServiceName="*") Action=REQ_BLOCKED* CEF_Severity>=0 src=*) OR ( sourcetype=cisco:asa message_id=302013 OR message_id=302015 OR message_id=302020  Cisco_ASA_action=allowed src=* ) 
| fields sourcetype src, CEF_Name, Action, request, Cisco_ASA_action, dest_ip, dest_port
| stats dc(sourcetype) as sts values(CEF_Name) as CEF_Name values(Action) as Action values(request) as request values(Cisco_ASA_action) as Cisco_ASA_action values(dest_ip) as dest_ip values(dest_port) as dest_port by src 
| where sts=2 | fields - sts

lightech1
Path Finder

Thank for your response but we need to get the events that have a src that appears on both sourcetypes.
With your search, we get all the events not only the ones that have the same src on both sourcetypes.

Thank's again.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Almost forgot about that. Try the updated answer now.

0 Karma

lightech1
Path Finder

Thanks!!! It works perfectly.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

You're using two of the most expensive command, transaction and join. can you list down the field name in each sourcetype that you want to see in the final output?

0 Karma

lightech1
Path Finder

Thanks for your reply,

for the sourcetype= incapsula:
src, CEF_Name, Action, request

for the cisco sourcetype:
Cisco_ASA_action, dest_ip, dest_port

Thanks!!

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...