Splunk Search

Can you help me improve the performance of a custom search?

att35
Builder

Hi,

We have Linux Auditd data coming into Splunk with sourcetype=linux:audit. In Auditd logs, Record Types define events based on what information is being presented, e.g. a type of "CWD" is triggered to record current working directory, type "SYSCALL" records a system call to kernel etc. Full list here in case anyone's interested. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/security_guide/sec-aud...

To find out when any user runs a potentially risky command e.g. rm -rf XYZ or chmod 777 XYZ, I use the following search which joins three types(CWD, EXECVE and SYSCALL) based on msg field which holds the id unique across all types for a specific Event.

sourcetype=linux:audit type=CWD 
| fields + msg, cwd
| JOIN msg [ search sourcetype=linux:audit  NOT auid=4294967295 type=SYSCALL comm=chmod OR comm=rm OR comm=chown
| fields + _time, msg, auid]
| JOIN msg [ search sourcetype=linux:audit NOT auid=4294967295 type=EXECVE (a0=chmod (a1=-R a2=777 OR a2=755) OR (a1=777)) OR (a0=rm a2=-r OR a2=-rf) OR (a0=chown)    | fields + _time, msg, host, a0, a1, a2, a3]  | table _time, host, msg, auid, a0, a1, a2, a3, cwd

Search works fine and gets the results but it takes exceptionally long to execute and almost every time I see the following under job summary.

info : [subsearch]: Search Processor: Subsearch produced 50000 results, truncating to maxout 50000.
warn : The limit has been reached for log messages in info.csv. 20 messages have not been written to info.csv. Please refer to search.log for these messages or limits.conf to configure this limit.
Is the search missing out on some data? Is there a way to make search more efficient and ensure it searches every event in the given time range?

Any helps is appreciated..

~ Abhi

0 Karma

bhavikbhalodia
Path Finder

Hi abhijittikekar,

Try to run below query.

sourcetype=linux:audit type=CWD
| table msg, cwd
| map
[ search sourcetype=linux:audit NOT auid=4294967295 type=SYSCALL comm=chmod OR comm=rm OR comm=chown msg=$msg$
|eval cwd=$cwd$
| table _time, msg, auid,cwd]
|map

[ search sourcetype=linux:audit NOT auid=4294967295 msg=$msg$ type=EXECVE (a0=chmod (a1=-R a2=777 OR a2=755) OR (a1=777)) OR (a0=rm a2=-r OR a2=-rf) OR (a0=chown)

| eval auid=$auid$ , cwd=$cwd$
| table + _time, msg, host, a0, a1, a2, a3,auid,cwd]]
| table _time, host, msg, auid, a0, a1, a2, a3, cwd

Thanks,
Bhavik

0 Karma

woodcock
Esteemed Legend

Like this:

(index=YouShouldAlwaysSpecifyIndexValues AND sourcetype=linux:audit) AND 
((type="CWD")
OR
(NOT auid="4294967295" AND type="SYSCALL" AND (comm="chmod" OR comm="rm" OR comm="chown"))
OR
(NOT auid="4294967295" AND type="EXECVE" AND (a0="chmod" AND (a1="-R" AND (a2="777" OR a2="755")) OR (a1="777")) OR (a0="rm" AND a2="-r*") OR (a0="chown")))
| fields _time msg, cwd, auid, host, a0, a1, a2, a3
| stats values(*) AS * BY msg
0 Karma

valiquet
Contributor

index=?? sourcetype=linux:audit type=CWD OR ( index= ?? sourcetype=linux:audit NOT auid=4294967295 type=SYSCALL comm=chmod OR comm=rm OR comm=chown or (INDEX=?? sourcetype=linux:audit NOT auid=4294967295 type=EXECVE (a0=chmod (a1=-R a2=777 OR a2=755) OR (a1=777)) OR (a0=rm a2=-r OR a2=-rf) OR (a0=chown) )

| fields + _time, msg, auid, cwd,host, a0, a1, a2, a3]
| stats values(*) as * by msg

0 Karma

nrduren1115
Explorer

You can use stats commands to accomplish the same thing:

| stats values(field1) as field1, values(field2) as field2 by msg
0 Karma

skoelpin
SplunkTrust
SplunkTrust

Watch this video, apply what you learned and your search will be significantly faster

https://conf.splunk.com/files/2018/recordings/master-joining-datasets-without-fn1784.mp4

0 Karma

att35
Builder

Thanks for the reference. There is tons of information there.. going through the content. i'll re post updated search after making changes.

~ Abhi

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...