How do I get the outer values in a transaction that has repeated startswith endswith parameters?

[nmulm]

Hi there,

I've been trying to solve an issue I have when using transactions. Here's an example of the logs I am working with ~

** ^0-15 only there for clarity\illustration not in actual logs

0 [9/26/15 0:31:06:105 CDT] 0000958c Mad I classmad info Begin|txntype|196476||||011|0123456789|0123456789
1 [9/26/15 0:31:06:105 CDT] 0000958c SystemOut O Begin|txntype|196476||||011|000016914100015|0123456789
2 [9/26/15 0:31:06:105 CDT] 0000958c SystemOut O Other row
3 [9/26/15 0:31:06:105 CDT] 0000958c SystemOut O Other row
4 [9/26/15 0:31:06:105 CDT] 0000958c SystemErr O Other row
5 [9/26/15 0:31:06:106 CDT] 0000958c SystemOut O End|txntype|196476||2|87|003|0123456789|0123456789|011|0123456789|0123456789
6 [9/26/15 0:31:06:108 CDT] 0000958c Mad I classmad info End|txntype|196476||1|848|003|0123456789|16057
7 [9/26/15 0:58:02:332 CDT] 0000013a Mad I classmad info Begin|txntype|221183||||011|0123456789|0123456789
8 [9/26/15 0:58:02:332 CDT] 0000013a SystemOut O Begin|MEMGET|221183||||011|000000762100300|99945750000172
9 [9/26/15 0:58:02:342 CDT] 0000013a SystemOut O Other rows
10 [9/26/15 0:58:02:372 CDT] 0000013a SystemInfo O Other rows
11 [9/26/15 0:58:02:373 CDT] 0000013a Mad I classmad info End|txntype|221183||2|65|011|0123456789|0123456789|011|0123456789|0123456789

The fields 0000958c\0000013a represent a thread id which should be present throughout the transaction and can be reused once a transaction finishes. The strings Mad\SystemOut represent the classes generating the logging, while the third pipe delimited field is a unique correlation id which only appears in lines with Begin \ End, for rows 0,1,5 & 6 for example this value is 196476.

I am interested in capturing the transaction around the outer rows containing Begin and End strings by thread and correlation id.
In that vein, rows 0-6 should be a transaction based on startwith "Begin|" and Endswith "End|"
as should rows 7-11.

Unfortunately the logging order of these classes Mad\SystemOut is not guaranteed, it is also not guaranteed that both will log a Begin\End line, but it is guaranteed that between them a Begin & End will be logged so we could end up with logs like this just as validly as the one's above ~

12 [9/26/15 0:31:06:105 CDT] 0000958c SystemOut O Begin|txntype|196464||||011|000016914100015|0123456789
13 [9/26/15 0:31:06:105 CDT] 0000958c SystemWarn O Other row
14 [9/26/15 0:31:06:108 CDT] 0000958c Mad I classmad info End|txntype|196464||1|848|003|0123456789|16057
15 [9/26/15 0:31:06:106 CDT] 0000958c SystemOut O End|txntype|196464||2|87|003|0123456789|0123456789|011|0123456789|0123456789

in which case I would expect the transaction to be rows 12-15.

I have been using the following transaction, but am getting inconsistent results as I do not feel I am getting the outer rows for Begin and End based on the Thread.

| transaction correlationid thread startswith="Begin|" endswith="End|"

Is there anyway to force this to use the widest match possible i.e. the outer matching cases. Also I know that by using the correaltionid I only get the begin \ end lines, but I couldn't see how to perform the match without having it in there. I've also tried using various combinations of the class type with Begin\End but it was no use as you are never sure exactly which you are going to get.

Thanks in advance,
N

[DalJeanis]

Try this...

| sort 0 _time thread 
| streamstats last(correlationid) as mycorrelation by thread   
| transaction mycorrelation thread 

[nmulm]

Thanks for getting back to me, I couldn't quite get that to work but it did start me down another path that works. So basically although I only have the 'correlation id' at the start and end I create a field and artificially populate it with the correlation id.

Logic is
- If line has a correlation id do nothing, use it
- If line has no correlation id and thread is the same as previous thread, use the previous correlation id
- If line has no correlation id and thread is NOT the same as previous thread, ignore it

Seems to be working reasonably well so far, basic query is ~

eval t=time | search host=|
sort thread, host, _time |

eval myCid=coalesce(correlationid,prevCorrelationid,t+""+thread+"unknown") |
streamstats current=f window=1 global=f last(myCid) as prevCid |
streamstats current=f window=1 global=f last(thread) as prevThread |
eval myCid=if(thread=prevThread,coalesce(correlationid,prevCid,t+""+thread+"unknown"),coalesce(correlationid,t+""+thread+"_unknown"))