Getting Data In

How to remove all events containing specific values in Splunk?

Hemnaath
Motivator

Hi All, Can any one guide me, on how to remove all events containing only the below listed events from rest of the events ? I am sure that we need to configure props.conf and transforms.conf , but not sure what to configure in props and how to right a regex in transforms.conf to remove the events .

Event Timestamp: April 24, 2017 12:54:34

Event Timestamp: April 24, 2017 12:55:30
Event Timestamp: April 24, 2017 12:56:34

Kindly guide me how to configure to remove the above events from rest of the events.

0 Karma
1 Solution

Richfez
SplunkTrust
SplunkTrust

Firstly, do you actually need to *delete* the events from the indexes? If that's the case, then...

Carefully craft a search that returns those rows and ONLY those rows. There's isn't enough information here to know precisely what it is that search will look like. Maybe index=X sourcetype=Y "Event Timestamp: April 24, 2017 12:56:34" OR "Event Timestamp: April 24, 2017 12:54:34" OR "Event Timestamp: April 24, 2017 12:55:30" Make double sure this works right. Then follow the steps in the documentation on removing data from indexes using YOUR search to prevent those results from showing up again.

If instead you only need to make sure you don't index those when they come in, you'll have to be more specific about why you want them gone. Much like the problem above where I had to say I don't know precisely what it is you are trying to delete, I don't know exactly what it is you are trying to remove. Are these empty lines (basically) with just a timestamp? Where are they coming from? How do they relate to the log lines you WANT to keep? Best would be to remove them from the sending source (or configure your syslog collector or whatever to ignore them). We can go down this route if necessary (and probably someone will!)

View solution in original post

bogdan_nicolesc
Communicator

Hi there!

I've got something similar with this:

source="WinEventLog:Security" | eval EventCode=if(EventCode="4801","NewName4801",EventCode) | eval EventCode=if(EventCode="4800","NameNew4800",EventCode) | eval EventCode=if(EventCode="4625","CertainDescription4625",EventCode)

But now i want to remove some EventCode from Field List in order to remain only 4801, 4800, 4625.

How do i do that? What do i have to type next in serch tab?

Thank you.

0 Karma

bogdan_nicolesc
Communicator

Found my answer from here: link text

So i had to make a list of codes to be removed from searchresult and enter it in search bar:

source="WinEventLog:Security" Type=* NOT (EventCode=4719 OR EventCode=4624 OR EventCode=4672 OR EventCode=4627 OR EventCode=4634 OR EventCode=4648 OR EventCode=4688 OR EventCode=4616 OR EventCode=4826 OR EventCode=4957 OR EventCode=4776 OR EventCode=1100 OR EventCode=4902 OR EventCode=4647 OR EventCode=1101 OR EventCode=4696 OR EventCode=4905 OR EventCode=4904) | eval EventCode=if(EventCode="4801","Deblocat4801",EventCode) | eval EventCode=if(EventCode="4801","NewName4801",EventCode) | eval EventCode=if(EventCode="4800","NameNew4800",EventCode) | eval EventCode=if(EventCode="4625","CertainDescription4625",EventCode)

0 Karma

Hemnaath
Motivator

Hi Somesoni, thanks I have modified and used the above props and transforms.conf in Heavy Forwarder instances to remove the above events and keep the rest of the events to be indexed.

Modified Transforms.conf

[setnull]
REGEX = Event Timestamp
DEST_KEY = queue
FORMAT = nullQueue

After implementing the transforms.conf, the events are removed and could see other events getting indexed now.

0 Karma

Richfez
SplunkTrust
SplunkTrust

Firstly, do you actually need to *delete* the events from the indexes? If that's the case, then...

Carefully craft a search that returns those rows and ONLY those rows. There's isn't enough information here to know precisely what it is that search will look like. Maybe index=X sourcetype=Y "Event Timestamp: April 24, 2017 12:56:34" OR "Event Timestamp: April 24, 2017 12:54:34" OR "Event Timestamp: April 24, 2017 12:55:30" Make double sure this works right. Then follow the steps in the documentation on removing data from indexes using YOUR search to prevent those results from showing up again.

If instead you only need to make sure you don't index those when they come in, you'll have to be more specific about why you want them gone. Much like the problem above where I had to say I don't know precisely what it is you are trying to delete, I don't know exactly what it is you are trying to remove. Are these empty lines (basically) with just a timestamp? Where are they coming from? How do they relate to the log lines you WANT to keep? Best would be to remove them from the sending source (or configure your syslog collector or whatever to ignore them). We can go down this route if necessary (and probably someone will!)

Richfez
SplunkTrust
SplunkTrust

To recap "how to delete data" from the docs, it's basically
1) Add "delete" capabilities to a role (preferably use a special user)
2) Log in as that role
3) Create a search that returns the events you want gone. Like mysearchhere
4) mysearchhere | delete
5) Confirm.

0 Karma

Hemnaath
Motivator

Hi rich, actually we are getting the below list of events along with other useful events containing detailed information, so we wanted to remove the below un-wanted events from being indexed in to splunk.

search query : index=x sourcetype = xxx:xxx:xxx

output:
4/24/17
1:44:41.000 PM

Event Timestamp: April 24, 2017 13:44:41
host = test.xxx.com source = /opt/syslogs/network/test.xxx.com/xxx.log sourcetype = xxx:xxx:xxx
4/24/17
1:44:41.000 PM

Event Timestamp: April 24, 2017 13:44:41
host = test.xxx.com source = /opt/syslogs/network/test.xxx.com/xxx.log sourcetype = xxx:xxx:xxx
4/24/17
1:44:41.000 PM

Event Timestamp: April 24, 2017 13:44:41
host = test.xxx.com source = /opt/syslogs/network/test.xxx.com/xxx.log sourcetype = xxx:xxx:xxx

So to discard the above events and keep the rest of the actual events, I had created a props.conf and transform.conf but not sure whether above configuration will help in removing the data.
Kindly guide in doing in this.

0 Karma

Hemnaath
Motivator

Hi All, can any one guide me on whether the below props and transforms.conf can be used to remove the unwanted data from getting into splunk. i.e is to discard specific events and keep the rest in splunk.

Discard specific events and keep the rest events

props.conf

[source::/opt/syslogs/network/.../xxx.log] --- > Source of the file from where splunk reads the data
TRANSFORMS-null= setnull

Transforms.conf

[setnull]
REGEX = [Event Timestamp: April 24, 2017 12:54:34] -- > events to be removed from the data
DEST_KEY = queue
FORMAT = nullQueue

thanks in advance

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Seems to be correct and inline with what Splunk suggests. (http://docs.splunk.com/Documentation/Splunk/6.5.2/Forwarding/Routeandfilterdatad#Discard_specific_ev...). Make sure it's setup in the Splunk instance which does the parsing (Indexer or heavy forwarder, whichever comes first in the flow) and restart Splunk.

0 Karma

Hemnaath
Motivator

Hi All, I have configured props.conf and transforms.conf to discard specific events and keep the rest events in splunk. But not sure whether this is the correct regex to do so. Kindly help in correcting the regex and setting it right to remove specific events and keep the rest events indexed.

Discard specific events and keep the rest events

props.conf

[source::/opt/syslogs/network/.../xxx.log] --- > Source of the file from where splunk reads the data
TRANSFORMS-null= setnull

Transforms.conf

[setnull]
REGEX = [Event Timestamp: April 24, 2017 12:54:34] -- > events to be removed from the data
DEST_KEY = queue
FORMAT = nullQueue

Kindly guide me whether the above props and transforms.conf can be used to remove specific events and keep the rest.

0 Karma

Richfez
SplunkTrust
SplunkTrust

I just answered - this comment is very helpful and I wish we wouldn't have been both updating at the same time. Still, the answer I gave on deleting them may be useful to others. 🙂

So, do you need to not index these events, or do you need to delete the events already indexed?

The answer I gave below should help with actually deleting them. If you need to not index them then it will require a different answer, much more like your technique above.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...