How can I not index commented lines from my files?

[Rob]

How can I avoid having lines that are commented within my files from being indexed by Splunk?

Lets say I have a log file that includes the following:

# Welcome to my file
# The lines with a # should be ignored by Splunk
12:34:56.123 Begin log file
12:34:58.123 Another event

How can I filter out the first two lines to prevent them from being indexed?

[nithish]

I had similar issue and nothing worked for me.

Below props and transforms worked for me in my scenario where it ignored the lines started with #:

transforms.conf:

[setnull]
REGEX=^[#\r\n]
DEST_KEY=queue
FORMAT=nullQueue

props.conf:

[sourcetype]
TRANSFORMS-ignore_comments = setnull

[deepamshah]

the above dint work for me.

This is what works in our environment

props.conf
[sourcetypeName]
TRANSFORMS-ignoreCommentedLines = ignoreLinesWithHash

[ignoreLinesWithHash]
REGEX = ^(?:[#].*)
DEST_KEY = queue
FORMAT = nullQueue

[Rob · Accepted solution]

This can be done by adding the appropriate lines to your props.conf and transforms.conf files in order to make sure that only uncommented lines are indexed.

Given the above example, here is what you would want to have...

props.conf

[iis]
TRANSFORMS-noComments = ignoreComments

transforms.conf

[ignoreComments]
REGEX = ^[^#.*$].*$
DEST_KEY = queue
FORMAT = indexQueue

This will then ignore any line beginning with the # character and index the rest of the file.

This means if you like you can choose to replace whatever the # in the above regex with whatever comment character your lines begin with. Just be sure to escape any regex syntax characters with a backslash ().

You can also use this same regex in your Splunk searches (rex or regex commands) such as:

...| regex _raw="^[^#.*$].*$"

Also, you may wish to take a look at the alternate method (for sending IIS commented logs to the nullQueue) listed here:
http://splunk-base.splunk.com/answers/9313/ignoring-comments-in-log-files

[nealpotter]

I downvoted this post because regex doesn't work and the wrong format is specified if you want to ignore events. it should go to nullqueue

[Takajian]

You need to configure props.conf and transforms.conf.

In your props.conf:

[your_sourcetype]
TRANSFORMS-xxx_log = setnull

In your transforms.conf:

[setnull]
REGEX = ^#.*\
DEST_KEY = queue
FORMAT = nullQueue

You can also refer to following ansower.

http://splunk-base.splunk.com/answers/11617/route-unwanted-logs-to-a-null-queue

[kjycls]

thank you for your kindness.