How can I search and alert for recurring events?

[aramakrishnan]

I'm trying to write a search which can detect the occurrence of an event AFTER a previous event containing the same field value has occurred. The use case is that when the first event occurs, we trigger an alert (which has a subsequent course of action internally) and when the subsequent log event occurs matching the same ID, we know that the issue has been fixed.

Event 1: <time=11/2/2017 11:00:00> sourcetype=firstevent "Event 1 happened"  devID=ABCD | ...
Event 2: <time=11/3/2017 02:00:00> sourcetype=secondevent "Event 2 happened"  deviceID=ABCD | ...

Ideally, we'd like for Splunk to search for the occurrence of the second event from the time the first event occurred. So for every device ID in event 1, look for a subsequent event 2 since the first event occurred, and trigger an alert saying "for this device ID, the second event has happened".

Couple of notes:
- The device ID field name is different in the two events, I have previously corrected it using an eval, for example:

search < event1> | eval deviceID =  devID

• The second event always occurs AFTER the first event. And the second event is basically indicating that a device has been registered, we have a thousand registrations a day, and it's hard to specify a time modifier. So either I could go with something generic and say "search in the last 2 weeks if you found a registration for devID" or I could say "search from earliest="when the first event happened" to now for the second event"

Things I've tried so far:

event 1 OR event 2 | eval deviceID =  devID |  transaction deviceID | event 1 AND event 2

I tried to put both events in a transaction and say only show me results where BOTH events are available so I can alert on that. However, this doesn't seem to work as expected perhaps because of the eval to change the field name?

Also, to specify time range. I tried putting the timestamp of event 1 into a field (say "first_event_start") using eval and then used a join to look for the second event starting earliest=first_event_start, but I got an error (looks like it only takes numeric values or time modifiers).

Any help would be great! Thanks!

[horsefez]

Hi,

have you tried using transaction with the startswith and endswith parameters yet?

Try something like this:

<yoursearch> | transaction deviceID startswith="<indicator that event 1 happened>" endswith="<indicator that event 2 happened>"

The indicator I'm talking about is some sort of raw string that marks the event

[Sukisen1981]

hi have you tried the streamstats command?