Getting Data In

Reqex filter in transforms.conf not working

working_dog
Explorer

At the indexer, I am trying to exclude event records from incoming windows logs that have Logon Type=3. Below is the configuration that I have, but doesn't seem to work.

$SPLUNK_HOME/etc/system/local/props.conf

[WinEventLog:Security]    
TRANSFORMS-security= events-null, events-null3, events-filter

$SPLUNK_HOME/etc/system/local/transforms.conf

[events-null]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[events-null3]
REGEX=Logon Type=\s*(3)\D
DEST_KEY = queue
FORMAT = nullQueue

[events-filter]
REGEX=(?msi)^EventCode=(4624|4634|4672|4768|4769|4776)\D
DEST_KEY = queue
FORMAT = indexQueue

The [events-filter] stanza works. I only see the events listed. The [events-null3] doesn't work. I've tried a variety of regex variations. Many came from different questions posted on this forum. Here are the ones I've tried:

REGEX=Logon_Type=\s*(3)\D
REGEX=Logon Type=\s*(3)\D
REGEX=Logon_Type=\s*(3)
REGEX=(?msi).Logon\sType:\s3D
REGEX=(?msi).Logon\sType:\s3
REGEX=(?m).*Logon\sType:\s+3.*
REGEX=(?m).Logon\sType:\s+3
REGEX = (?msi).*Logon Type:\s*(2|10)

Below is a sample of an event I am trying to filter out.


12/02/2013 08:45:43 AM
LogName=Security
SourceName=Microsoft Windows security auditing.
EventCode=4634
EventType=0
Type=Information
ComputerName=P-UMS03.p-umhs.med.umich.edu
TaskCategory=Logoff
OpCode=Info
RecordNumber=5744631540
Keywords=Audit Success
Message=An account was logged off.

Subject:
Security ID:        P-UMS\USPCNTBSA1$
Account Name:       USPCNTBSA1$
Account Domain:     P-UMS
Logon ID:       0xb71e31b1

Logon Type:         3

Thanks for any insight you can provide

--Mike

Tags (1)
1 Solution

cphair
Builder

I believe the third regex from the bottom in your list should work. I got a similar one to match from the search interface with rex (don't have a convenient props.conf to test with at the moment):


rex ".(?<foo>Logon Type:\s+\d)."

That said, the transforms stanzas in props.conf are applied in the order they're listed. In your case, the logon type 3 events match both events-null3 and events-filter. Are you sure directing to nullqueue takes effect and cancels out the events-filter action? Try removing events-filter (or swapping the order, though that may have the same problem) and see if the logon type 3 events are dropped properly.

View solution in original post

working_dog
Explorer

Thank you for your tips, they were helpful. I swapped the order as you suggested and was able to filter out Logon Types after that. But I still had an error with my regex that was causing all Login Types to be filtered out instead of ony those with a value of 3. I finally got that fixed, here is my final transforms.conf:

$SPLUNK_HOME/etc/system/local/transforms.conf

[events-null]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[events-filter]
REGEX=(?msi)^EventCode=(4624|4634|4672|4768|4769|4776)\D
DEST_KEY = queue
FORMAT = indexQueue


[events-null3]
REGEX=(?m)^Logon\sType:\s+3\D
DEST_KEY = queue
FORMAT = nullQueue

cam343
Path Finder

In version 6.4 this configuration (props.conf & transforms.conf) needed to be applied on the Universal Forwarder not the indexers. Hopefully that saves some people some time.

MuS
SplunkTrust
SplunkTrust

@cam343 - this is only valid for structured data http://docs.splunk.com/Documentation/Splunk/6.4.0/Data/Extractfieldsfromfileswithstructureddata#Forw... and not for all kind of data.

cphair
Builder

I believe the third regex from the bottom in your list should work. I got a similar one to match from the search interface with rex (don't have a convenient props.conf to test with at the moment):


rex ".(?<foo>Logon Type:\s+\d)."

That said, the transforms stanzas in props.conf are applied in the order they're listed. In your case, the logon type 3 events match both events-null3 and events-filter. Are you sure directing to nullqueue takes effect and cancels out the events-filter action? Try removing events-filter (or swapping the order, though that may have the same problem) and see if the logon type 3 events are dropped properly.

working_dog
Explorer

Thank you for your tips, they were helpful. I swapped the order as you suggested and was able to filter out event types after that. But I still had an error with my regex which I finally nailed down. Here is my final transforms.conf:

$SPLUNK_HOME/etc/system/local/transforms.conf

[events-null]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[events-filter]
REGEX=(?msi)^EventCode=(4624|4634|4672|4768|4769|4776)\D
DEST_KEY = queue
FORMAT = indexQueue


[events-null3]
REGEX=(?m)^Logon\sType:\s+3\D
DEST_KEY = queue
FORMAT = nullQueue
0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...