Our configuration has universal forwarder - so the whole log file is being forwarded to the indexer. I know there is lot of data/information/warning in the log file which is not required for monitoring. How can I ignore those data so that indexer don't have to index so much (not needed) data ? I understand there is some configuration for that - but couldn't figure out from the deployment document of splunk. Or am I looking at the wrong document ?
You do that through props.conf and transforms.conf on the indexer, since that is where the parsing takes place. If you hava a full/heavy forwarder, you can do the operation there.
For instance, if your special log file contains a lot of events containing WARNING, and you don't want to index them, your config should look something like;
In props.conf
[source::/var/your_special.log]
TRANSFORMS-set= setnull
In transforms.conf
[setnull]
REGEX=WARNING
DEST_KEY=queue
FORMAT=nullQueue
Also see;
http://www.splunk.com/wiki/Where_do_I_configure_my_Splunk_settings
http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf
http://docs.splunk.com/Documentation/Splunk/latest/Admin/Transformsconf
Hope this helps,
Kristian
You do that through props.conf and transforms.conf on the indexer, since that is where the parsing takes place. If you hava a full/heavy forwarder, you can do the operation there.
For instance, if your special log file contains a lot of events containing WARNING, and you don't want to index them, your config should look something like;
In props.conf
[source::/var/your_special.log]
TRANSFORMS-set= setnull
In transforms.conf
[setnull]
REGEX=WARNING
DEST_KEY=queue
FORMAT=nullQueue
Also see;
http://www.splunk.com/wiki/Where_do_I_configure_my_Splunk_settings
http://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf
http://docs.splunk.com/Documentation/Splunk/latest/Admin/Transformsconf
Hope this helps,
Kristian