How to make splunk interpret multiple lines as single line

[c0mrade]

I have a pretty long log that needs to be analyzed, not single lined though, here is example #1:

.....some unimportant data many lines...
2011-11-07 13:05:48,060 INFO com.mysoftware.log.splunk.test.Processor: loadStatus = NEW
2011-11-07 13:05:47,984 INFO com.mysoftware.log.splunk.test.Processor: DELTA status determined for record with ID: 010bbd25aeccaacb564fab543c5b0429083c804a 
2011-11-07 13:05:47,984 INFO com.mysoftware.log.splunk.test.Processor: DELTA status: ID=bebbe8570c4ce87238378b53241a976a5528dfaf, {TEST:DELTA_STATUS=NEW, TEST:JOB_ID=job_201110281559_2009, TEST:LAST_UPDATED_TIMESTAMP=Mon, 7 Nov 2011 13:05:47, TEST:ROW_ID=010bbd25aeccaacb564fab543c5b0429083c804a, TEST:LOAD_JOB_ID=job_201110281559_2008} 
2011-11-07 13:05:48,060 INFO com.mysoftware.log.splunk.test.Processor: loadStatus = NEW
2011-11-07 13:05:48,060 INFO com.mysoftware.log.splunk.test.Processor: DELTA status determined for record with ID: 0141daa46fa7a576b538d2437a339f8ad041f0b7 
2011-11-07 13:05:48,060 INFO com.mysoftware.log.splunk.test.Processor: DELTA status: ID=ecaf46341bb7c040ece713e87f0308308093838c, {TEST:DELTA_STATUS=NEW, TEST:JOB_ID=job_201110281559_2009, TEST:LAST_UPDATED_TIMESTAMP=Mon, 7 Nov 2011 13:05:48, TEST:ROW_ID=0141daa46fa7a576b538d2437a339f8ad041f0b7, TEST:LOAD_JOB_ID=job_201110281559_2008} 
2011-11-07 13:05:48,130 INFO com.mysoftware.log.splunk.test.Processor: loadStatus = NEW
2011-11-07 13:05:48,130 INFO com.mysoftware.log.splunk.test.Processor: DELTA status determined for record with ID: 01b9bba9474f3c838931242883a7462722fb45b1 
2011-11-07 13:05:48,130 INFO com.mysoftware.log.splunk.test.Processor: DELTA status: ID=c9711472457d8f4b226c75da5a9ce2cfee099680, {TEST:DELTA_STATUS=NEW, TEST:JOB_ID=job_201110281559_2009, TEST:LAST_UPDATED_TIMESTAMP=Mon, 7 Nov 2011 13:05:48, TEST:ROW_ID=01b9bba9474f3c838931242883a7462722fb45b1, TEST:LOAD_JOB_ID=job_201110281559_2008} 

Here is example of how one line/record should look like, example #2:

2011-11-07 13:05:48,060 INFO com.mysoftware.log.splunk.test.Processor: loadStatus = NEW
2011-11-07 13:05:47,984 INFO com.mysoftware.log.splunk.test.Processor: DELTA status determined for record with ID: 010bbd25aeccaacb564fab543c5b0429083c804a 
2011-11-07 13:05:47,984 INFO com.mysoftware.log.splunk.test.Processor: DELTA status: ID=bebbe8570c4ce87238378b53241a976a5528dfaf, {TEST:DELTA_STATUS=NEW, TEST:JOB_ID=job_201110281559_2009, TEST:LAST_UPDATED_TIMESTAMP=Mon, 7 Nov 2011 13:05:47, TEST:ROW_ID=010bbd25aeccaacb564fab543c5b0429083c804a, TEST:LOAD_JOB_ID=job_201110281559_2008} 

So this example #1 should be read as 3 records(3 lines), not multiple lines as usual. I went trough some splunk documents realized that I need to add a new source type and define line breaking for myself.

And in my etc/system/local added props.conf and inside defined new source type and added } as a LINE_BREAKER but it's not working it seems that it breaks my line by default when it sees the timestamp. How do I do this? How do I make splunk treat these 3 separate lines as one?

[tgow]

You might want to experiment with the SHOULD_LINEMERGE. If you want to use the LINE_BREAKER then you need to add the following:

SHOULD_LINEMERGE = true

[kristian_kolb · Accepted solution]

I think that would be very hard to do, since there is nothing in the separate lines of an 'event' that bind them together, not even the time stamps (if I understand you correctly.)

Your best bet is probably to use the transaction command in a rather inefficient manner, if the first line to be considered part of the event contains "loadStatus = NEW" and the last line contains the the string "DELTA staus:" (note the comma).

In order for transaction to work, the lines need to be 'grouped' on a common piece of information. Since there are no ID values that are common to the event, you'll have to find one an make sure it won't change.

Given the example log you provided, you can extract the log level, e.g. INFO, or the "com.mysoftware..." string and run the transaction on that.

... | transaction log_level startswith="loadStatus = NEW" endswith="DELTA status:"

Please note that this will (partly) fail if:
a) log lines are not written sequentially, and only one 'event' at a time.
b) 'events' do contain alternate values for the transaction 'group-by' variable

For more information on the transaction command, see the docs.

Ideally, you should have the logging software write a transaction ID which would make this whole operation much simpler.

UPDATE: if you just want to break the lines after the '}' you could always edit your props.conf

[your_sourcetype]
SHOULD_LINEMERGE = false
LINE_BREAKER = }([\r\n]+)
...

However, this still requires that the lines are written sequentially, one 'event' at a time.

hope this helps,

kristian