I would like to create a search that is first able to determine when one or more incidents or events have occurred. Then from that search, I would like to see all other events that have occurred within a specified time range around each of those incidents.
I'm guessing the search will follow something like this, but I don't have a good idea on how to actually create it:
search for each [sub search for key expression - returns set of event times] earliest=-5m@m latest=+5m@m | transaction etc...
Any help or examples would be great! Thanks
Here is an example:
sourcetype=mystuff [ search sourcetype=mystuff "interesting event" | head 1 | eval earliest=_time-5 | eval latest=_time+5 | fields earliest,latest | format "(" "(" "" ")" "OR" ")" ]
This will find the single most recent "interesting event"
in the myfirst
sourcetype, and then display everything for 5 seconds around that in the same sourcetype.
Comments:
You don't really need head 1
, and
this search will work fine without
it. If there are multiple
"interesting event"
s, you'll get
everything withing 5 seconds of any
of them.
The format
command is actually a
hack because there's something that I
consider to be a bug in the search
syntax. If I file that and it gets
fixed, you won't need that clause.
Here is an example:
sourcetype=mystuff [ search sourcetype=mystuff "interesting event" | head 1 | eval earliest=_time-5 | eval latest=_time+5 | fields earliest,latest | format "(" "(" "" ")" "OR" ")" ]
This will find the single most recent "interesting event"
in the myfirst
sourcetype, and then display everything for 5 seconds around that in the same sourcetype.
Comments:
You don't really need head 1
, and
this search will work fine without
it. If there are multiple
"interesting event"
s, you'll get
everything withing 5 seconds of any
of them.
The format
command is actually a
hack because there's something that I
consider to be a bug in the search
syntax. If I file that and it gets
fixed, you won't need that clause.
Simpler:
sourcetype=mystuff [ search sourcetype=mystuff "interesting event" | eval earliest=_time-5 | eval latest=_time+5 | return earliest latest]
I tried to be greedy by not stating the source, but the search then seemed to ignore the OR'd time ranges. The inner search was still just as quick because of the
Thanks! You're awesome. I had to think about what that format is doing and a way to have multiple time ranges (within reason). Below is the search I ended up with.
[search index=<> source=<>.
http://docs.splunk.com/Documentation/Splunk/5.0/SearchReference/Localize
Generates a list of time contiguous event regions defined as: a period of time in which consecutive events are separated by at most 'maxpause' time. The found regions can be expanded using the 'timeafter' and 'timebefore' modifiers to expand the range after/before the last/first event in the region respectively. The Regions are return in time descending order, just as search results (time of region is start time). The regions discovered by localize are meant to be feed into the map command, which will use a different region for each iteration. Localize also reports: (a) number of events in the range, (b) range duration in seconds and (c) region density defined as (#of events in range) divided by (range duration) - events per second.
Example 1: Search the time range of each previous result for "failure".
... | localize maxpause=5m | map search="search failure starttimeu=$starttime$ endtimeu=$endtime$"
Example 2: As an example, searching for "error" and then calling localize finds good regions around where error occurs, and passes each on to the search inside of the the map command, so that each iteration works with a specific timerange to find promising transactions
error | localize | map search="search starttimeu::$starttime$ endtimeu::$endtime$ |transaction uid,qid maxspan=1h"
Thanks, we'll have use for this in the near future. In this use case, the user actually needs to read the interactions between our back office and partners.