Splunk Search

hide the duplicate events

rbw78
Communicator

Hello,

I have 2 sources of events with "almost" the same framework and some of them reference the same event with the same content.
I want to hide in my search the events which matchs between the both sources for seeing only the single events.
I don't know what kind of function i have to use to do that (dedup, correlate ????)

Here is a single event from the 2 differents sources

Event in source 1

Jan  5 14:11:09 10.233.118.0 Jan  5 14:11:09 LX2AW8DSY CEF:0|Trend Micro|OfficeScan Intrusion Defense Firewall|6.1.0.69|1004371|Mozilla Firefox Obfuscated URLs Within Iframes Vulnerability|6|dmac=F0:DE:F1:18:58:49 smac=00:21:55:EF:8F:FF src=10.5.69.50 dst=10.233.118.0 in=1414 proto=TCP spt=8080 dpt=63034 cs2=0x00 ACK cnt=1 act=IDS:Log cn3=503 cs5=761296 cs1=Obfuscated_URL_i cs6=8

Event in source 2

1/5/2012 14:11,LX2AW8DSY,DPI Rule: 1004371 - Mozilla Firefox Obfuscated URLs Within Iframes Vulnerability,Web Client Mozilla FireFox,Detect Only: Log,25,Reverse Flow,F0:DE:F1:18:58:49,TCP,ACK,10.5.69.50,00:21:55:EF:8F:FF,8080,10.233.118.0,F0:DE:F1:18:58:49,63034,1400,"""Obfuscated URL i""",0,0,,1,

I know i can use the date, hours, minutes and type of event to check if there's a duplicate, doing a dedup of the 2 events and hide it.

How could i do that ?

thanks

Tags (3)
0 Karma

rtadams89
Contributor

Dedup is still going to return one of the events though. If you want to return only those events that are not in both sets, you could:

Perform a search taht returns both event types in the same result set. Add a field to each event that identifies what "type" or source each event comes from. Maybe | eval type1=if(index=1, "TRUE", "FALSE") | eval type2=if(index=2, "TRUE", "FALSE") |. Then use the transaction command to merge similar events into a single event. Finally, add | where NOT (type1=TRUE AND type2=TRUE) to the end of the search string.

Ayn
Legend

Use dedup with the fields you want to dedup on as arguments. So for instance if you have a field rule in both events and want to filter multiple events within the same day of the month, hour and minute, you would do:

... | dedup rule date_mday date_hour date_minute

EDIT: Sorry, I see now that I misunderstood your question, I didn't realize you wanted to remove BOTH events. For that I would advise you to use transaction in a similar way:

... | transaction rule maxspan=1m | search eventcount=1

This will remove all transactions which have more than one event (i.e. the ones with duplicate messages).

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...