Index selected events of log file

[carljohan]

I have a log file namned: wrapper.log
This log file has two different type of events defined with the prefix INFO or ERROR.
I want to index only the ERROR events but am not getting it to work.
Im on a mac.

Here is my log file:

ERROR | jvm 1 | 2013/05/03 10:47:52 Test_error

INFO | jvm 1 | 2013/05/03 10:48:52 Test

ERROR | jvm 1 | 2013/05/03 10:49:52 Test_error

INFO | jvm 1 | 2013/05/03 10:50:52 Test

ERROR | jvm 1 | 2013/05/03 10:51:52 Test_error

INFO | jvm 1 | 2013/05/03 10:52:52 Test

ERROR | jvm 1 | 2013/05/03 10:53:52 Test_error

inputs.conf:

[monitor:///Users/carljohan/logs/wrapper.log]

disabled=false

sourcetype = ESB_Wrapper

props.conf:

[ESB_Wrapper]

SHOULD_LINEMERGE=false

LINE_BREAKER = ([\r\n]+)[0-9]+-[0-9]+-[0-9]+\s+

TIME_FORMAT = %Y/%m/%d %H:%M:%S,%3N

TRANSFORMS-set= setnull,setparsing

tranfsforms.conf

[setnull]

REGEX = .

DEST_KEY = queue

FORMAT = nullQueue

[setparsing]

REGEX = (\W|^)ERROR(\W|$)

DEST_KEY = queue

FORMAT = indexQueue

With this setup all events are still being indexed.
What am I doing wrong?

[lguinn2 · Accepted solution]

I think that your LINE_BREAKER may be causing part of the problem. You should not need a LINE_BREAKER if you have set SHOULD_LINEMERGE to false

props.conf:

[ESB_Wrapper]
SHOULD_LINEMERGE=false
TIME_FORMAT = %Y/%m/%d %H:%M:%S,%3N
TRANSFORMS-set= setnull,setparsing

Finally, this REGEX should be sufficient.

REGEX = ^ERROR

You don't need to match the whole event. So a trailing '.*' is unnecessary and more expensive.

ONE MORE THING:

Where are your props.conf and transforms.conf? They need to be located where the data is parsed - usually the indexer(s). And, you will need to restart the indexer(s) to make these changes effective. And, your parsing changes are not retroactive - they will only affect new data as it is parsed.

[linu1988]

why not ERROR.+ only?

[somesoni2]

Mind trying this for REGEX

"(?m)^ERROR.*"

[lukejadamec]

Run a search that pulls the logs listed above, and test the regex like this:
| regex ^ERROR
It should only show log entries that start with ERROR. If it does not, adjust the regex.

[carljohan]

They are indexed on the local splunk instance. No forwarders are included in the setup.

[Ayn]

Are you indexing these events on a local Splunk instance or are you forwarding these from your machine to a separate indexer?

[carljohan]

I tried with ^ERROR which gives me the following transforms.conf and restarted Splunk but it did not work.

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = ^ERROR
DEST_KEY = queue
FORMAT = indexQueue

[lukejadamec]

I'm not a regex wizard, but I should think ^ERROR should work. You will need to restart splunkd on the indexer for the change to take effect.

[carljohan]

I posted the complete .conf content.
What should I change in the REGEX?

[lukejadamec]

The configs look good except for the REGEX. Is what you posted missing characters?