Splunk Search

How can I search and alert for recurring events?

aramakrishnan
New Member

I'm trying to write a search which can detect the occurrence of an event AFTER a previous event containing the same field value has occurred. The use case is that when the first event occurs, we trigger an alert (which has a subsequent course of action internally) and when the subsequent log event occurs matching the same ID, we know that the issue has been fixed.

Event 1: <time=11/2/2017 11:00:00> sourcetype=firstevent "Event 1 happened"  devID=ABCD | ...
Event 2: <time=11/3/2017 02:00:00> sourcetype=secondevent "Event 2 happened"  deviceID=ABCD | ...

Ideally, we'd like for Splunk to search for the occurrence of the second event from the time the first event occurred. So for every device ID in event 1, look for a subsequent event 2 since the first event occurred, and trigger an alert saying "for this device ID, the second event has happened".

Couple of notes:
- The device ID field name is different in the two events, I have previously corrected it using an eval, for example:

search < event1> | eval deviceID =  devID
  • The second event always occurs AFTER the first event. And the second event is basically indicating that a device has been registered, we have a thousand registrations a day, and it's hard to specify a time modifier. So either I could go with something generic and say "search in the last 2 weeks if you found a registration for devID" or I could say "search from earliest="when the first event happened" to now for the second event"

Things I've tried so far:

event 1 OR event 2 | eval deviceID =  devID |  transaction deviceID | event 1 AND event 2

I tried to put both events in a transaction and say only show me results where BOTH events are available so I can alert on that. However, this doesn't seem to work as expected perhaps because of the eval to change the field name?

Also, to specify time range. I tried putting the timestamp of event 1 into a field (say "first_event_start") using eval and then used a join to look for the second event starting earliest=first_event_start, but I got an error (looks like it only takes numeric values or time modifiers).

Any help would be great! Thanks!

Tags (2)
0 Karma

horsefez
SplunkTrust
SplunkTrust

Hi,

have you tried using transaction with the startswith and endswith parameters yet?

Try something like this:

<yoursearch> | transaction deviceID startswith="<indicator that event 1 happened>" endswith="<indicator that event 2 happened>"

The indicator I'm talking about is some sort of raw string that marks the event

0 Karma

Sukisen1981
Champion

hi have you tried the streamstats command?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...