Getting Data In

Filtering events using NullQueue

Michael_Schyma1
Contributor

I was wondering if there is any way to filter eventcodes, but not every event that is being passed through. For example is there a way to block EventCode 4624, but just the debug messages and let the rest pass?

This is what we currently have to block windows EventCodes:

REGEX=EventCode=(4624|4776|4662|4634|4688|4648|4907|4768|4624||538|560|552|534)

We want to remove EventCode=4624 leaving the rest. the EventCode=4624 is generated because of "An account was successfully logged on" event on all servers. We want this enabled for most windows servers, but want to block this event from our 13 domain controller hostnames. Is it possible to have multiple regexes sending to null Queues?

1 Solution

lguinn2
Legend

[Updated to show that you can do multiple transforms]

Okay - given the answer from Kristian about the type, I think I can show you how to filter the events. Assuming that the sourcetype is called WinEventLog:Security...

props.conf

[WinEventLog:Security]
TRANSFORMS-t1=eliminate-4624-debug
TRANSFORMS-t2=eliminate-eventcodes

transforms.conf

[eliminate-4624-debug]
REGEX=(?m)EventCode\s*=\s*4624.*?Type\s*=\s*Debug\s
DEST_KEY=queue
FORMAT=nullQueue

[eliminate-eventcodes]
REGEX=EventCode=(4776|4662|4634|4688|4648|4907|4768|4624||538|560|552|534)
DEST_KEY=queue
FORMAT=nullQueue

Now, this is not the tightest regular expression, so I would test it with the following search:

sourcetype="WinEventLog:Security" 
| regex _raw="(?m)EventCode\s*=\s*4624.*?Type\s*=\s*Debug\s"

If this search matches only the data that you want to eliminate, then great. Otherwise, I may still need to see a sample of the data...

View solution in original post

shayfa
Path Finder

Hi Guys,

can i do the same as : REGEX=EventCode=(4776|4662|4634|4688|4648|4907|4768|4624||538|560|552|534)

if the EventCode is a field i created in an extract field ?

0 Karma

lguinn2
Legend

[Updated to show that you can do multiple transforms]

Okay - given the answer from Kristian about the type, I think I can show you how to filter the events. Assuming that the sourcetype is called WinEventLog:Security...

props.conf

[WinEventLog:Security]
TRANSFORMS-t1=eliminate-4624-debug
TRANSFORMS-t2=eliminate-eventcodes

transforms.conf

[eliminate-4624-debug]
REGEX=(?m)EventCode\s*=\s*4624.*?Type\s*=\s*Debug\s
DEST_KEY=queue
FORMAT=nullQueue

[eliminate-eventcodes]
REGEX=EventCode=(4776|4662|4634|4688|4648|4907|4768|4624||538|560|552|534)
DEST_KEY=queue
FORMAT=nullQueue

Now, this is not the tightest regular expression, so I would test it with the following search:

sourcetype="WinEventLog:Security" 
| regex _raw="(?m)EventCode\s*=\s*4624.*?Type\s*=\s*Debug\s"

If this search matches only the data that you want to eliminate, then great. Otherwise, I may still need to see a sample of the data...

jotne
Builder

Some comments to the above post.

Its better to remove stuff at the Universal Forwarder instead of HF or Index.

So to remove 4662, add the following to an input.file

# Used to block 4662 message 
 
[WinEventLog://Security]
blacklist1 = 4662

 

Or you can do like this.  Block all 4662 message except 4662 with Message="ms-Mcs-AdmPwd"

[WinEventLog://Security]
whitelist1 = EventCode="^4662$" Message="ms-Mcs-AdmPwd"
whitelist2 = EventCode="^((?!4662$)[0-9]*)$"

 

 

 
Take care with this:
REGEX=EventCode=(4776|4662|4634|4688|4648|4907|4768|4624||538|560|552|534)

This may block all, due to the double ||,   I gess that is a typo.
Also it will block 1552, 5525 etc, so here you should use ^ and $

0 Karma

lguinn2
Legend

@erstexas - it depends. On a Universal Forwarder, no. On a heavy forwarder, yes, you can place the transforms.conf and the props.conf on the forwarder.

However, Splunk generally recommends that you use a Universal Forwarder and do this parsing on the indexers. This keeps the processing load low on the production server that is running the forwarder. If you are thinking that you want to limit the network traffic, good idea but - experience says that it isn't worth the trouble unless you will be eliminating more than 50% of the events before forwarding.

0 Karma

erstexas
Path Finder

Can this be placed on the servers that are running the Forwarder? I would rather have it not sent to the Indexer at all. Or maybe that is what is implied?

0 Karma

lguinn2
Legend

Michael - I would suggest that you have different stanzas in props.conf that invoke different stanzas in transforms.conf

All of the stanzas could send data to the nullqueue but each would have a different regex. Even if there is a way to combine them, I would probably keep them separate for clarity.

0 Karma

wrangler2x
Motivator

I am curious how you would change using nullqueue/blacklist to the more common way of doing it where you have a "pass" transform with a whitelist and nullqueue anything else, but still allow for the special case he brought up here where you want to drop the debug 4624 events.

0 Karma

Michael_Schyma1
Contributor

This is what we currently have to block windows EventCodes:

REGEX=EventCode=(4624|4776|4662|4634|4688|4648|4907|4768|4624||538|560|552|534)

We want to remove EventCode=4624 leaving the rest. the EventCode=4624 is generated because of "An account was successfully logged on" event on all servers. We want this enabled for most windows servers, but want to block this event from our 13 domain controller hostnames. Is it possible to have multiple regexes sending to null Queues?

0 Karma

Michael_Schyma1
Contributor

I believe this is exactly what i am looking for. We were just trying to use the debug messages as an example to get the concept. I will test it out next week and let you know. I thank you very much.

0 Karma

kristian_kolb
Ultra Champion

well, I made an assumption regarding the Type=Debug... I'd also like sample data...

/k

0 Karma

kristian_kolb
Ultra Champion

If you by debug mean the Type=Debug (I don't know if it exists, I only have 'Informational' in my logs). Therefore I used ComputerName in the example below.

The following regex works with rex inline in the search. It should probably work with the instructions in http://docs.splunk.com/Documentation/Splunk/4.3.1/Deploy/Routeandfilterdatad#Discard_specific_events...

sourcetype=wineventlog:security EventCode="4624" | rex ".*(?<blaha>EventCode=4624[\n\r\w\.=]*ComputerName=some.host.name).*"

Hope this helps,

Kristian

Michael_Schyma1
Contributor

This is what we currently have to block windows EventCodes:

REGEX=EventCode=(4624|4776|4662|4634|4688|4648|4907|4768|4624||538|560|552|534)

We want to remove EventCode=4624 leaving the rest. the EventCode=4624 is generated because of "An account was successfully logged on" event on all servers. We want this enabled for most windows servers, but want to block this event from our 13 domain controller hostnames. Is it possible to have multiple regexes sending to null Queues?

0 Karma

Michael_Schyma1
Contributor

Thank you for your help Kristian.

0 Karma

lguinn2
Legend

To use the nullQueue, you must be able to write a regular expression that identifies the events to be eliminated.

For the event code, that would be

EventCode\s*=\s*4624

but I am not sure how you would identify this as a debug message. Can you post an example of a few events?

kristian_kolb
Ultra Champion

Yes you can have multiple transforms that send stuff to the null queue;

props.conf
[sourcetype_x]
TRANSFORMS-delete_stuff = drop_a, drop_b

transforms.conf
[drop_a]
REGEX = a
DEST = queue
FORMAT = nullQueue

[drop_b]
REGEX = b
DEST = queue
FORMAT = nullQueue

That's the same as having REGEX = a|b in one nullQueue transform.

0 Karma

joesrepsolc
Communicator

What would my REGEX line in the transforms.conf be to ELIMINATE any events that don't have this string? I must be missing something. I only want to ingest events that have this string at the beginning of the line:  "|>>>>>>|" 

In REGEX that should be ^\|>>>>>>\| right? 

So how to i set the transforms.conf REGEX= line to say anything that doesn't match the above REGEX, drop to the nullqueue?

 

Thanks in advance!!!

Joe

0 Karma

Michael_Schyma1
Contributor

I dont know if that is exactly what i was looking for. I probably worded the question in a confusing way.

Heres another example:
we want to remove EventCode=4624 leaving the rest. the EventCode=4624 is generated because of "An account was successfully logged on" event on all servers. We want this enabled for most windows servers, but want to block this event from our 13 domain controller hostnames. Is it possible to have multiple regexes sending to null Queues? AND if so how would we do this?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...