Search for Events that Happened Around the time of Other Events that Also meet other Criteria

[Kyle_Brandt]

How do I make a query that will search for events that happened around the same time as the results of another query?

So for example, if I search for "host='foo' bar", how can I find other events that happened within say 5 seconds of any "host='foo' bar" events?

Secondly, how would I modify that search so it returns events that happened within 5 seconds of "host='foo' bar" events that also match a query such as "baz"? So only events within 5 seconds of "host='foo' bar" that have the string "baz" in them (while still including the "host='foo' bar" events.

Hope I am being clear, these are basically a sort of event correlation searches and I imagine are pretty basic, I just can't find the magic Google keywords.

[BenjaminWyatt]

This may not be exactly what you're after, but it's worth mentioning that the "Power Actions" app (http://apps.splunk.com/app/1547/) implements some of this functionality in the workflow - you can search +- 1 minute or +- 2 seconds around any event. That's not as robust as what you can do with yannK's approach, of course, but it's easy to implement for people who don't know the Splunk search language.

[yannK]

I found a much better method
using the wonderful FORMAT default option.

• the main search is your second search
• the sub search is your first search to find the event time and the other parameters the time will be used to define a time range of 120 seconds around each events. the results are formatted as OR/AND conditions using FORMAT.

Example to find the ip of a server with ERROR from an index into another index, 2 minutes around.

sourcetype=myothersourcetype  [ search ERROR

| eval earliest=_time-120 | eval latest=_time+120 | fields src_ip earliest latest | FORMAT "(" "(" "" ")" "OR" ")" ]

Here is a simple search to find events 120 seconds around a splunk restart

index=_internal  [ search index=_internal sourcetype="splunkd" "(build" 

| eval earliest=_time-120 | eval latest=_time+120 | fields host earliest latest | FORMAT "(" "(" "" ")" "OR" ")" ]

[yannK]

If you are looking for the classic : show me all events 10 minutes around midnight last night, you can use this version.

index=_internal [|stats count| eval _time=now() | bucket _time span=1d
| eval earliest=_time-60 | eval latest=_time+60 | table earliest latest | FORMAT "(" "(" "" ")" "OR" ")"]

[bbingham]

I think I understand what you're asking, by default splunk does something a little different with events "time", run your search "host='foo' bar" then after the events are listed, simply click the time on the event line you'd like to search around (the date next to the gray pull-down menu in the actual event line). This will then set the search time to the specified time you clicked on. Splunk I believe has a 30 second default window for you to "drill" down on. After your search time is constrained, then simply type back in your search line the new search you'd like to look for, such as "baz"

Hope this helps!

[bbingham]

After you zoom in to a selected time frame, there is a "zoom out" button that tries to intelligently expand a time frame. Try giving that a shot!

[Kyle_Brandt]

I did find that, but that isn't really ideal for seeing patterns. Always better to have it in front of you. But while we are on that, can you manually increase that 30 second windows somehow?