Hi,
I am a newbie with a task to implement a monitoring functionality on Splunk. The requirement is for Splunk to be able to monitor an application 's live logs where each line in the logs have a format with multiple fields such as timestamp.
Each entry is an event that occurs in the application being monitored.
Each event is conceptually grouped together as a processing session that occurs in the application by virtue of the same file these events are operating on. There is no built in/native session Id like in HttpSession. (the application being monitored is not a web application).
Each session can consist of multiple events.
There are events that must come in pairs (Event A occurs and must be followed with Event B within 5 minutes). Otherwise, an email must be sent to notify the on-call by email about this.
From my understanding of Splunk so far, the best way I can think of is to implement is to:
Group events into transactions since that is the best way for Splunk to handle a session.
Create an alert that checks each event that must be in pairs in the transactions is paired with the arrival of the second event must occur within 5 minutes. Otherwise, sends an email.
The alert runs the search every 1 minute for events happening within the last 24 hours
If so, how do I handle the following requirements:
How do provide the on-call a way to tell Splunk that the offending events raised by the alert is being handled. Hence, don't keep sending an email? Is there a way to create a screen on Splunk to handle this?
How about if an offending event cannot be handled within 24 hours, how do I make Splunk forgets about the event will fall outside the 24 hour coverage of the alert's search? How do I store the list of the offending events in Splunk? in otherwords, how Splunk maintains a state for an alert?
Sorry for the long post and I thank you ahead for any help!
Thanks
... View more