Getting Data In

How to configure selective data indexing ?

somnathnag
Engager

Our configuration has universal forwarder - so the whole log file is being forwarded to the indexer. I know there is lot of data/information/warning in the log file which is not required for monitoring. How can I ignore those data so that indexer don't have to index so much (not needed) data ? I understand there is some configuration for that - but couldn't figure out from the deployment document of splunk. Or am I looking at the wrong document ?

Tags (2)
0 Karma
1 Solution

kristian_kolb
Ultra Champion

You do that through props.conf and transforms.conf on the indexer, since that is where the parsing takes place. If you hava a full/heavy forwarder, you can do the operation there.

For instance, if your special log file contains a lot of events containing WARNING, and you don't want to index them, your config should look something like;

In props.conf

[source::/var/your_special.log]
TRANSFORMS-set= setnull

In transforms.conf

[setnull]
REGEX=WARNING
DEST_KEY=queue
FORMAT=nullQueue

Also see;

http://www.splunk.com/wiki/Where_do_I_configure_my_Splunk_settings

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Transformsconf

Hope this helps,

Kristian

View solution in original post

0 Karma

kristian_kolb
Ultra Champion

You do that through props.conf and transforms.conf on the indexer, since that is where the parsing takes place. If you hava a full/heavy forwarder, you can do the operation there.

For instance, if your special log file contains a lot of events containing WARNING, and you don't want to index them, your config should look something like;

In props.conf

[source::/var/your_special.log]
TRANSFORMS-set= setnull

In transforms.conf

[setnull]
REGEX=WARNING
DEST_KEY=queue
FORMAT=nullQueue

Also see;

http://www.splunk.com/wiki/Where_do_I_configure_my_Splunk_settings

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Transformsconf

Hope this helps,

Kristian

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...