How to use subsearch to only return server events that are NOT followed by a recovery event within a certain period of time?

[lennys26]

I have a search that returns server events and would like to know when this event is NOT followed by a recovery message within a short period of time.

In my example below, the event is triggered and it recovers 5 seconds later. In this case, I would NOT want this to return results.

Oct 21 06:40:13 cam-vm-mon3 mfsmount[3425]: master: connection lost (1)
Oct 21 06:40:18 cam-vm-mon3 mfsmount[3425]: registered to master

To give my question better context, I am actually running this in an old Splunk 4.3 install and am configuring this as an alert, so these are alert criteria more than search criteria

I have been playing with append and appendpipes but am not having any luck. Can anyone offer any suggestions?

[woodcock · Accepted solution]

Like this (with a base search that limits to only 2 types of events):

... | eval type=case(searchmatch("master: connection lost"), "down", searchmatch("registered to master"), "up", true(), "BUG!") | reverse | streamstats count(eval(type="down")) AS sessionID by host | eventstats latest(_time) AS latestTime latest(type) AS latestType by sessionID  host | where type="down" | eval downSeconds=if(latestType="down", now(), latestTime) - _time | where downSeconds > 5

[lennys26]

Additional thoughts on paths forward.

• Search for "master". Eval/search for "registered". Where this returns NULL, return details from Master search (or return something!), else, return NULL.

OR

• In the past I have played (unsuccessfully) with the idea of simply using a counter to....say.... run a search for "master" and a sub-search for "registered". For each event matching "master" add 1 (to x), for each "registered" subtract 1. Then where x>0, do something