How to merge multi-line messages to one event in search query

[chrismok]

Hi All

We want to index multiline log messages with no timestamp as one event.

But regular expression for multiline is difficult.

So now I try following configurations in props.conf.

[Test_buildmasterlog]
SHOULD_LINEMERGE = false
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%z
EXTRACT-bid = (?m)^BuildID:(?P<bid>.+)^Release:
EXTRACT-release = (?m)^Release:(?P<release>.+)[\r\n]
EXTRACT-environment = ^Environment:(?P<environment>.+)^BuildType:
EXTRACT-buildtype = (?m)^BuildType:(?P<buildtype>.+)
EXTRACT-starttime = (?m)^Start Time:(?P<starttime>.+)^Status:
EXTRACT-status = (?m)^Status:(?P<status>.+)^Elapsed Time:
EXTRACT-elapsedtime = (?m)^Elapsed Time:(?P<elapsedtime>.+)

And the log is look like that

BuildID:20140818-00
Release:R20.5
Environment:QA3
BuildType:Daily Auto Build
Start Time:2014-08-18 11:00:00 000
Status:Completed
Elapsed Time:25mins

However when I do a searching using the following query

sourcetype="test_buildmasterlog"|table bid,release,environment, buildtype, starttime, status,elapsedtime

It will separate to two rows. So which place that I set wrongly?

http://oi61.tinypic.com/2mr9r4g.jpg

[lguinn2 · Accepted solution]

Your props.conf is specifically telling Splunk that your events are 1 line each. That's not what you want! Instead of this

[Test_buildmasterlog]
 SHOULD_LINEMERGE = false
 TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%z

Try this

[Test_buildmasterlog]
 SHOULD_LINEMERGE = false
BREAK_ONLY_BEFORE = ^BuildID
 TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%z

The rest of props.conf is fine.

[chrismok]

Thanks Lguinn. It's work.
And one more question. Look at your conf, you define only the keyword of "BuildID" for break the line only. if I don't know which field is the first appear in the log. How can I do?

[grijhwani]

One would expect entries in the same log file to follow a standard pattern. The "break_only_before" is predicated on that assumption. There is only so much one can achieve with configurations. To some extent it is necessary for the input source to conform to some reasonable standard of expected behaviour.

[adityapavan18]

@Chrismok: As from your props file i dont see any configuration that tells splunk to index multiple lines together.
Could you please provide log sample.