hello MuS, How to search these events that meet "3 same contents and their positions is sequent in 2 seconds"

[wangweibee]

How to search these events that meet the condition of "3 same contents(except time message) in 2 seconds", give me a idea, if possible, please.

our logs have timestamps in front. for example:
the original file:

"
Oct 31 2014 14:31:11 xiaowang ;Last diagnostic message repeated 14 times.
Oct 31 2014 14:34:11 xiaowang %%01INFO/4/SUPPRESS_DIAGLOG(D):CID=0x80600406;
Oct 31 2014 14:34:11:012 xiaowang %%01INFO/4/SUPPRESS_DIAGLOG(D):CID=0x80600406;
Oct 31 2014 14:34:12 xiaowang %%01INFO/4/SUPPRESS_DIAGLOG(D):CID=0x80600406;
Oct 31 2014 14:35:22 xiaowang %%01INFO/4/SUPPRESS_DIAGLOG(D):CID=0x80600406;
Oct 31 2014 14:34:32 xiaowang %%01RMBASE/6/PARTNER_FSM_TRANSFER(D):CID=80; 
Oct 31 2014 14:34:56 xiaowang %%01DEBUG/4/DBG_TOP_PROC_MEM(D);
"

we hope to get these logs that have the same content(except the timestamps in front), but the time info in front may be different and the time interval is within 2 second. if we can find 3 logs that meet the frontal condition, we record it.

the expected result is:

"
Oct 31 2014 14:34:11 xiaowang %%01INFO/4/SUPPRESS_DIAGLOG(D):CID=0x80600406;
Oct 31 2014 14:34:11:012 xiaowang %%01INFO/4/SUPPRESS_DIAGLOG(D):CID=0x80600406;
Oct 31 2014 14:34:12 xiaowang %%01INFO/4/SUPPRESS_DIAGLOG(D):CID=0x80600406;
"

this question block me very long time, How, if at all possible, could this be accomplished?
attention: the time info format of logs may be different, for example: "Oct 31 2014 14:34:11" and "Oct 31 2014 14:34:11:012".
I want to 'transaction' and 'top' to deal with the problem, but I cannot devide a log item into two parts, one is timestamps in front and the other is log main content after the time info.

any advice?

[MuS · Accepted solution]

Hi wangweibee,

the timestamp should not be a problem at all for Splunk, but you should get your content into a field first; try this:

your base search here | rex "\s\w+\s(?<myFoo>%.+)" | ... 

This creates a new field called myFoo (assuming all your interesting events start with a % ) which can be used in any further search command. So add a transaction to it and your done:

your base search here | rex "\s\w+\s(?<myFoo>%.+)" | transaction "myFoo"

If this is what you want, use the regex and setup automatic field extraction http://docs.splunk.com/Documentation/Splunk/6.2.0/Knowledge/Createandmaintainsearch-timefieldextract... so you would not need the rex in your search.

hope this helps ...

cheers, MuS

[wangweibee]

thanks! "rex" can solve it.
but, now I need to search consecutive and same logs. that is, In addition to meet "3 same contents(except timestamps in front) in 2 seconds", the events must sequent in position, for example:

the original log:
"
Oct 31 2014 14:34:08 repeated 114 times.
Oct 31 2014 14:34:11 xiaowang CID=0x80600406; repeated 4 times.
Oct 31 2014 14:34:11 xiaowang CID=0x80600406; repeated 4 times.
Oct 31 2014 14:34:11 repeated 114 times.
Oct 31 2014 14:34:11 xiaowang CID=0x80600406; repeated 4 times.
Oct 31 2014 14:34:15 xiaowang CID=0x80600406; repeated 4 times.
Oct 31 2014 14:34:22 xiaowang CID=0x80600406; repeated 4 times.
Oct 31 2014 14:34:24 xiaowang CID=0x80600406; repeated 4 times.
"

the hindermost 4 events meet the criteria, and we get the result:
"
Oct 31 2014 14:34:11 xiaowang CID=0x80600406; repeated 4 times.
Oct 31 2014 14:34:15 xiaowang CID=0x80600406; repeated 4 times.
Oct 31 2014 14:34:22 xiaowang CID=0x80600406; repeated 4 times.
Oct 31 2014 14:34:24 xiaowang CID=0x80600406; repeated 4 times.
"

how to realize it ? thanks for any advice!

this problem is urgent, please give me some help.
I want to create a new posting, but it I have not enough reputation.

[wangweibee]

thanks.
I need not timestamps that will is weeded. I need the content after the timestamps.
the regex of "rex field=_raw "\w{3} \d{2} \d{4} \d{2}:\d{2}:\d{2} (?.)" " can match a kind of timestamps format, but cannot match other time format.
And the regex of "rex field=_raw "_time (?.)"" does not work。
how to utilize the splunk timestamps field distilled method in rex expression ?

[MuS]

based on your provided examples use this to match everything after your events timestamp:

rex "\:\d{2,3}\s(?<myField>.*)"

[wangweibee]

thanks for all！
but, how to use regex to express the timestamps that have many format, such as "Oct 31 2014 14:34:11", '14/10/31 14:35:22.000' and "Oct 31 2014 14:34:11:012", and the content after timestamps have arbitrary format, and there are not delimiter between the timestamps and the hinder log content.
thanks!

[MuS]

no need to regex for timestamps, use _time as field. If your events have different formats and/or no delimiter you have to use these events in any online regex tester tool like www.regexr.com to get the matching regex.