Creating a search to return potentially unrelated events around a key events

[msorenson]

I would like to create a search that is first able to determine when one or more incidents or events have occurred. Then from that search, I would like to see all other events that have occurred within a specified time range around each of those incidents.

I'm guessing the search will follow something like this, but I don't have a good idea on how to actually create it:

search for each [sub search for key expression - returns set of event times] earliest=-5m@m latest=+5m@m | transaction etc...

Any help or examples would be great! Thanks

[gkanapathy · Accepted solution]

Here is an example:

sourcetype=mystuff [ search sourcetype=mystuff "interesting event" | head 1 | eval earliest=_time-5 | eval latest=_time+5 | fields earliest,latest | format "(" "(" "" ")" "OR" ")" ] 

This will find the single most recent "interesting event" in the myfirst sourcetype, and then display everything for 5 seconds around that in the same sourcetype.

Comments:

• You don't really need head 1, and this search will work fine without it. If there are multiple "interesting event"s, you'll get everything withing 5 seconds of any of them.

• The format command is actually a hack because there's something that I consider to be a bug in the search syntax. If I file that and it gets fixed, you won't need that clause.

[carasso]

Simpler:

sourcetype=mystuff [ search sourcetype=mystuff "interesting event" | eval earliest=_time-5 | eval latest=_time+5 | return earliest latest]

[msorenson]

I tried to be greedy by not stating the source, but the search then seemed to ignore the OR'd time ranges. The inner search was still just as quick because of the , but the outter appeared to scan the entire universe for that entire day.

[msorenson]

Thanks! You're awesome. I had to think about what that format is doing and a way to have multiple time ranges (within reason). Below is the search I ended up with.

[search index=<> source=<>. | dedup _raw | transaction source maxspan=5m | eval starttimeu=_time-60 | eval endtimeu=_time+300 | fields index,source,starttimeu,endtimeu | format "(" "(" "" ")" "OR" ")"] | dedup _raw | transaction source startswith="<>" maxspan=5m | search | reverse

[Dan]

Check out the "localize" search command.

http://docs.splunk.com/Documentation/Splunk/5.0/SearchReference/Localize

Description

Generates a list of time contiguous event regions defined as: a period of time in which consecutive events are separated by at most 'maxpause' time. The found regions can be expanded using the 'timeafter' and 'timebefore' modifiers to expand the range after/before the last/first event in the region respectively. The Regions are return in time descending order, just as search results (time of region is start time). The regions discovered by localize are meant to be feed into the map command, which will use a different region for each iteration. Localize also reports: (a) number of events in the range, (b) range duration in seconds and (c) region density defined as (#of events in range) divided by (range duration) - events per second.

Examples

Example 1: Search the time range of each previous result for "failure".

... | localize maxpause=5m | map search="search failure starttimeu=$starttime$ endtimeu=$endtime$"

Example 2: As an example, searching for "error" and then calling localize finds good regions around where error occurs, and passes each on to the search inside of the the map command, so that each iteration works with a specific timerange to find promising transactions

error | localize | map search="search starttimeu::$starttime$ endtimeu::$endtime$ |transaction uid,qid maxspan=1h"

[msorenson]

Thanks, we'll have use for this in the near future. In this use case, the user actually needs to read the interactions between our back office and partners.