Filtering data using SHOULD_LINEMERGE

[danillopavan]

Hi all,

I have configured the line breaking parameter as (SHOULD_LINEMERGE = true) to read a log file that contains the below data for each update. It will allow to have all data in the same event to run my searches:

S Doing: print 1597931/1
S --> Print Job @>SPOREQ:1597931@ S print job @>SPOREQ:1597931@ S replace user SAPSYS by NFEX
S print job @>SPOREQ:1597931@ S --------> db_rtab Error 128, table TSPEVJOB, action rspogdio_insert (expected)
S SpTSP01Select() - 1597931 is OTF/SMART/RDI/ADSx job, do not read TSP02L!
S Sß: (2017120211393600) sending job @>SPOREQ:1597931@DEV:JCD1@<'
S 1 processed

And I would like to filter the input data to index only the below lines of the log file disregarding the others lines:
S Doing: print 1597931/1
S replace user SAPSYS by NFEX

S Sß: (2017120211393600) sending job @>SPOREQ:1597931@DEV:JCD1@<'
S 1 processed

I was thinking to configure my REGEX just using the phases "Doing: print" , "replace user", "sending job", however it will not work for the line merge configuration as dont have the end of the line. So my question is: how I should configure the regex in transform file to allow just index 3 lines using SHOULD_LINEMERGE ?

I am using this below configuration on the props and transform files and it is not working:

[sourcetype]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE=processed
TRANSFORMS-set= setIndexnull,setIndexparsing

[setIndexnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setIndexparsing]
REGEX = print
DEST_KEY = queue
FORMAT = indexQueue

Thanks and regards,
Danillo Pavan

[danillopavan]

I still didn´t find any solution for this case. Still need to trash certain lines and then create multiline events out of what is left...
Any proposal??

[niketn]

@danillopavan this seems similar to other question your have posted: https://answers.splunk.com/answers/597389/filtering-data-using-should-linemerge.html

I would request you to consolidate required details against single question and keep only one of them open.

[danillopavan]

Any answer?

[sandyIscream]

Write your props and transforms.conf like the below and try this.

props.conf

[sourcetype]
TRANSFORMS-set = setnull, print1,print2,print3

Transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[print1]
REGEX = S\sDoing:
DEST_KEY = queue
FORMAT = indexQueue

[print2]
REGEX = S\sreplace
DEST_KEY = queue
FORMAT = indexQueue

[print3]
REGEX = S \sSß:
DEST_KEY = queue
FORMAT = indexQueue

[danillopavan]

Hello Sandy,

Now no lines are being indexed. The regex commands that you suggested seems to be wrong. Anyway, the instructions that you suggested, in my opinion, would not work. I need to conciliate MERGED LINES with FILTERING.

[danillopavan]

I am using this below configuration on the transform files and it is not working:

[setIndexnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setIndexparsing]
REGEX = print
DEST_KEY = queue

FORMAT = indexQueue