alert on "trigger2" only if not preceeded by "trigger1" up to 60 seconds before

[stucky101]

Splunk gurus

Newbie here trying to get a feel for what's possible in splunk. I'd like to alert on "trigger2"

Nov 4 19:34:00 blabla
Nov 4 19:34:01 trigger2
Nov 4 10:35:00 blabla

EXCEPT when it's preceded by "trigger1" up to 60s before "trigger2" like:

Nov 4 19:33:05 trigger1
Nov 4 19:34:00 blabla
Nov 4 19:34:01 trigger2
Nov 4 10:35:00 blabla

From what I've seen this means in splunk terms :

1. Search for "trigger2" but also output everything up to 60s before the matching line
2. Take this output and do a subsearch for "trigger1"
3. If found ignore but if not alert on "trigger2"

I see things like "localize" and "format" but I'm unsure if any of those can achieve this.
Splunk support says this is rather complicated. Anyone wants to take a stab ?
I'd love to see it.

[gkanapathy]

Not really. it's just a realtime search over 60 seconds, with

trigger1 OR trigger2
| stats latest(eval(if(searchmatch("trigger2"),_time,null()))) as t2time
        earliest(eval(if(searchmatch("trigger1"),_time,null()))) as t1time
| where isnotnull(t2time) AND (t1time <= t2time)

and alert if the number of results is greater than 0.

[gkanapathy]

well then it seems you could just alert when the number of results is 0.

[stucky101]

Thank you. I tried this in the dashboard with a 1 min realtime window but it results in the opposite behaviour.
When I log the 2 triggers within 60s of each other it shows results

t2time t1time

but the idea was that in this case it would not log anything since the 2 events showing up within 60 seconds imply a normal transaction.

It should trigger when only "trigger2" is found without a preceding "trigger1" up to 60s before. I tried that but this didn't alert at all.