Splunk Search

Search for Event "A" and NOT Event "B" within a given period of time

jodros
Builder

We are sending anti-virus logs to Splunk. I am trying to create a search that would first, find logs indicating event "A" which is a virus was detected, then look ahead 60 seconds for event "B" which is a virus was cleaned. I want to show only logs that find event "A" but NOT event "B". This basically are the events that we would need to manually correct. I also want to ensure that the computer and virus names the same in both log events. Below are some samples of the logs.

Any assistance would be appreciated. Thanks in advance.

01/14/2012 08:15:14 PM
LogName=Application
SourceName=AV Server
EventCode=600
EventType=3
Type=Warning
ComputerName=av-server1
User=SYSTEM
Sid=S-1-5-18
SidType=1
TaskCategory=System
OpCode=None
RecordNumber=17037
Keywords=Classic
Message=Virus/Malware: TROJ_GEN.R49C7AE
Computer: workstation01
File: C:\System Volume Information_restore{93998DF6-4B0C-40FB-9311-0B790FF41A3C}\RP292\A0028002.dll
Date/Time: 1/14/2012 20:15:14
Result: Virus detected

01/14/2012 08:15:45 PM
LogName=Application
SourceName=AV Server
EventCode=500
EventType=3
Type=Warning
ComputerName=av-server1
User=SYSTEM
Sid=S-1-5-18
SidType=1
TaskCategory=System
OpCode=None
RecordNumber=17038
Keywords=Classic
Message=Virus/Malware: TROJ_GEN.R49C7AE
Computer: workstation01
File: C:\System Volume Information_restore{93998DF6-4B0C-40FB-9311-0B790FF41A3C}\RP292\A0028002.dll
Date/Time: 1/14/2012 20:15:43
Result: Cleaned

Tags (3)
0 Karma
1 Solution

jodros
Builder

I was able to achieve what I needed, by using this thread's answer as a reference. The sanitized search I created is below. Thank you all for your assistance.

sourcetype="antivirus" [ search sourcetype="antivirus" EventCode=600 | eval earliest=_time | eval latest=_time+60 | fields earliest,latest | format "(" "(" "" ")" "OR" ")" ] | eval status = if(EventCode="500",1,0) | stats sum(status) as statusCount values(virus) by Computer,file_name | where statusCount = 0

This search finds an event indicating malware was found but not cleaned. It then uses that timestamp as a reference and look ahead 60 seconds. If it does not find an event indicating that the malware was cleaned, it will trigger. The search runs -6m@m to -1m@m every 5 minutes.

View solution in original post

jodros
Builder

I was able to achieve what I needed, by using this thread's answer as a reference. The sanitized search I created is below. Thank you all for your assistance.

sourcetype="antivirus" [ search sourcetype="antivirus" EventCode=600 | eval earliest=_time | eval latest=_time+60 | fields earliest,latest | format "(" "(" "" ")" "OR" ")" ] | eval status = if(EventCode="500",1,0) | stats sum(status) as statusCount values(virus) by Computer,file_name | where statusCount = 0

This search finds an event indicating malware was found but not cleaned. It then uses that timestamp as a reference and look ahead 60 seconds. If it does not find an event indicating that the malware was cleaned, it will trigger. The search runs -6m@m to -1m@m every 5 minutes.

kristian_kolb
Ultra Champion

Another option is to use a subsearch. Unfortunately there is no specific eventID that link the two events together (infected/cleaned), so this example will be less than perfect. Let's assume that the File field is unique enough for these puropses.

The main search is looking for "infected" events.
The subsearch returns the File field for all "cleaned" events.
This is then added to the main search with a NOT.
The result should be all "infected" events, minus those that have a corresponding "cleaned" event (based on the File field).

sourcetype=XXX EventCode=600 NOT [search sourcetype=XXX EventCode=500 | fields + File]

Replace the sourcetype, and add further qualifying fields (to both the subsearch and the main search) to get the correct results.

http://docs.splunk.com/Documentation/Splunk/latest/User/HowSubsearchesWork#Use_subsearch_to_correlat...


UPDATE:

The "timerange substituted" message is displayed because you alter the earliest/latest parameters within the search string, i.e. it differs from the dropdown-value (e.g. "last 15 min", "last 4 hrs" etc). The results you're getting may not be correct.

I don't think you should try to alter the time parameter (i.e. don't care if there is 60 seconds or so between the 500 and 600 events). Try instead to fine a common ID value for the two events, and use subsearch (as shown above) to find "infected" events that do not have a corresponding "cleaned" event.

Otherwise you should probably try transaction on a combination of fields like File, Virus/Malware, Computer etc. I don't know if a transaction is 'built' if there is only one event, i.e. it might not be a good to use startswith and endswith as described in another answer, since what you really want are the 'faulty' transactions, where there is no ending 'cleaned' event.
As Ayn stated, there is the eventcount parameter which could do the trick here.

Somebody more knowledgeable with transactions can maybe help out.

Hope this helps,

Kristian

0 Karma

southeringtonp
Motivator

FWIW, the use of endswith in transactions is a little counterintuitive. endswith=(EventCode=600) wouldn't actually require that a transaction contain an event 600. It just means that if an event 600 is found, it will be the last event in the transaction. The real point of using endswith for something like this would be to make sure that an event 500 will be recognized as a re-infection. If you have a Detected, then Cleaned, then Detected, you want to make sure that you don't report that as clean.

0 Karma

jodros
Builder

Kristian, I tried using your suggestion, with the file being the unique identifier between the events. I even tried Computer and a custom extraction virus. All return ONLY unable to clean events. However when I run a search just on the machine in the same timeframe, it also shows that within 60 seconds, there is a clean event for each unable to clean.

I do appreciate your assistance. I do not think that transactions will work, as there will never be an "ending" event. However I am not that familiar with transactions, so I am not sure if there is a way to make it work.

0 Karma

jodros
Builder

The search logic seems to be working. However the scheduled search found an event last night and sent an email. In the email, instead of only showing results from the last 5 minutes, it showed results for alltime. I then tried to run the search manually, and it gives me a notification saying, "Your timerange was substituted based on your search string." Why is it doing this?

Thanks

0 Karma

jodros
Builder

That was how I originally tried getting this to work. I finally think I have it working with this search:

sourcetype="antivirus" [ search sourcetype="antivirus" EventCode=600 | eval latest=_time+60 | fields + latest ] | eval status = if(EventCode="500",1,0) | stats sum(status) as statusCount values(virus) by Computer | where statusCount = 0

Does anyone notice any issues with this search? I don't want to run poorly crafted searches. I plan on running it -5m@m to now and schedule it to run every 5 minutes.

Thanks

0 Karma

dwaddle
SplunkTrust
SplunkTrust

You should look into transactions, which would be one way of making this work. Applicable docs are at http://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Abouttransactions

0 Karma

Ayn
Legend

eventcount may be better suited for checking how many events a transaction includes. 🙂

southeringtonp
Motivator

Usually there are two ways. Either look at the value of linecount (I said search linecount>1 before, but yours may be more than 1 since you're dealing with multi-line events. Or int this case a better way is probably to pipe to search again and put something like | search NOT EventCode=500 at the end instead of the search on linecount. (The transaction should leave you with a multivalue field for EventCode). There may also be a way to do it with streamstats and some clever search/where clauses -- would have to give that approach some more thought when I'm more awake.

jodros
Builder

If the transaction starts with eventcode=600 and "normally" ends with eventcode=500, how do I create a search that finds all events that start with eventcode=600 and fails to end with eventcode=500?

0 Karma

southeringtonp
Motivator

Definitely transactions. Using the 'startswith' option may help, as may the linecount field. Something like: SourceName="AV Server" EventCode=500 OR EventCode=600 | transaction Computer,Message startswith=(EventCode=500) endswith=(EventCode=600) | search linecount>1

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...