Splunk Search

REGEX format to remove unwanted log data using specific text with quotes and spaces

srunyon
New Member

I have a log that sends ( eventtype=dlp level=notice vd="PERIM" filteridx=0 filtertype=none filtercat=none severity=medium ) without the ( or ) and would like to know the regex format to use in transforms.conf to send it to the nullQueue. Any help would be greatly appreciated.

Tags (2)
0 Karma

DMohn
Motivator

If you really need to capture this exact event, you can use the following config:

props.conf:

 [your_sourcetype_stanza]
 TRANSFORMS-null1 = eliminate_events

transforms.conf

 [eliminate_events]
 REGEX=^\seventtype=dlp\slevel=notice\svd=\"PERIM\"\sfilteridx=0\sfiltertype=none\sfiltercat=none\sseverity=medium\s$
 DEST_KEY=queue
 FORMAT=nullQueue

You could still check if you may shorten the regex somehow, or make it mire flexible, if the events to be sent to nullQueue vary slightly.

0 Karma

srunyon
New Member

I see how the regex should work. I tried using both a [] and [source::] definitions but I am still getting the events. I can only guess that I am either placing the profs.conf and transforms.conf in the wrong location.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

So this is your actual raw data from the log file?/

eventtype=dlp level=notice vd="PERIM" filteridx=0 filtertype=none filtercat=none severity=medium 
0 Karma

srunyon
New Member

Yes, including the beginning and ending spaces.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

This Splunk doc should give you all the configurations that you need to configure for this event filtering.

http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Routeandfilterdatad#Discard_specific_e...

The regex could be ^\s*eventtype=dlp. Again, we're not sure if this will filter other events as well, so please provide some more samples of both the logs, one you want to keep and one you want to discard.

0 Karma

srunyon
New Member

To answer you point, I included only the part of the message that is consistent within the data that I do not want to index. This should ensure that I do not discard any valid messages.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Ok.. so based on where they (the pattern) appear in your data, use the appropriate REGEX. If your actual events starts with the pattern you posted, @DMohn's answer should do the trick. If they do not , remove the caret sign "^" from REGEX.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...