Splunk Search

How to filter a specific string from Splunk event logs at index-time?

snehalk
Communicator

Hello All,

I have requirement where need to filter(ignore) "---------------------------------------------" from the event logs, I have tried with the blacklist attribute in inputs.conf, but it is not working. Do we need to create props.conf and transforms.conf file, or we can ignore these dashes from inputs.conf only?

Please let me know which solution is best here, if we need to create a props.conf and transforms.conf, then what will be the contents of the files?

Thanks

0 Karma

santiagoaloi
Path Finder

Hey!

First off these configurations happen in the parsing phase, before the indexing and searching phase, therefore
both props.conf and transforms.conf should be placed in the indexer, not the search head as you mentioned before. My suggestion is that you place these files in $SPLUNK_HOME/etc/system/local

The sourcetype stanza in props.conf isn't right, it should be as follows:

props.conf

 [mydata]
 TRANSFORMS-null = setnull

Your regex is not matching the repeated dashes in the event, so I put together a little regex for you.

transforms.conf

 [setnull]
 REGEX      = ^.*\s(\-+\S)
 DEST_KEY   = queue
 FORMAT     = nullQueue

Restart Splunk after changing these files.

If you don't wanna have the wrong old indexed data with the repeated dashes, one option is to use the DELETE command in a search query that matches those events with "------" in them. i.e:

index=yourindex sourcetype=mydata "---------------------------------------------" | DELETE

This will hide the matched events from further searches but will not erase them completely from the index.

Splunk by default doesn't allow users to run the command DELETE, so go to "Access controls » Roles » "
and apply the role "can_delete" to your user.

Hope this works for you 😃

/Santiago

0 Karma

snehalk
Communicator

Hello Woodcock,

Thanks for reply, as suggested I have deployed props.conf and transforms.conf on search heads but still the events are not filtering please let me know where i went wrong.

Took sourcetype instate of source

Props.conf

[sourcetype::mydata]
TRANSFORMS-null= setnull

Transforms.conf

[setnull]
REGEX = [^----------------------------------------]
DEST_KEY = queue
FORMAT = nullQueue
0 Karma

woodcock
Esteemed Legend

You need to deploy these to your Indexers (not your Search Head) and then restart the Splunk instances running on them. After that, the new data will be fixed but the old/existing data will still be "wrong". Also, use this (no square brackets) instead of what you have:

REGEX = ^----------------------------------------

Don't forget to "Accept" and answer to close the question.

0 Karma

woodcock
Esteemed Legend

Did this work?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...