Receive 3 Errors - Same type within an Hour - Notify via Email

[suparn777]

Hello Guys,

I am setting up Splunk in such a way that i receive specific errors by email.
My path of folder is /var/log/apache/error.log

However i want to select it in such a way that if i receive 3 errors of same type within an hour only then i am notified of the error via email.

When i select source=/var/log/apache/error.log in Search and try to create an alert i do not find an option to fulfill my requirements.

Hence request your help in this scenario.

Thanks

[lguinn2]

Best guess, based on the information provided (and assuming that the apache_error sourcetype has a field called message)

sourcetype=apache_error | eventstats count by message | where count > 2 | sort _time message

Then "save as alert" and set the parameters to

• Alert Type: scheduled
• Run every hour at 5 minutes past the hour
  
• earliest=-1h@h (beginning of the prior hour)
• latest=@h (beginning of this hour)
• Trigger if number of results greater than zero

Here is a tutorial on alerts:

https://www.youtube.com/watch?v=SuARLqm7_jc

And a link to Define Scheduled Alerts in the Alerting Manual!

If there is no field called "message", then we will need to see a sample of your data to figure this out, as Apache log files are highly configurable.

[suparn777]

For example if i give the command:
index=hosts "[error]" AND perl

I get an error list containg the term perl. However i need to refine it such that when i receive 3 errors of same kind only then i am notified via email within an hour.

[suparn777]

Hello iguinn,

Thanks a lot for the answer. This seems to be working.However is it possible if we can modify this command so that it includes only commands which have the term perl in them? We want all errors related to Perl which are present in apache logs.
Also i am trying for a condition in which as soon as i receive 3 errors of same kind in an hour receive an email.
Is it possible to fulfill these requirements?

Thanks

[lguinn2]

Try this for the search

sourcetype=apache_error
| rex "\s-e\:\s(?<message>.*?\.)\s"
| eventstats count by message
| where count > 2
| sort _time message

[suparn777]

Also other logs are as follows:

2014-06-07T18:55:55-0700 [Sat Jun 07 18:55:55 2014] [error] [Sat Jun 7 18:55:55 2014] -e: Error executing run mode '
resp_questions_hier_viewmodel': Item already exists on another level at /usr/local/www/abc/perl/MenuFramework/
RespQuestionsHierViewModel.pm line 824.\n[Sat Jun 7 18:55:55 2014] -e: at /usr/local/www/abc/perl/
MenuFramework.pm line 298.\n
2014-06-07T18:55:00-0700 [error]-e: Error executing runmode resp_questions_hier_viewmodel': Item already exists on another level at /usr/local/www/abc/perl/MenuFramework/
RespQuestionsHierViewModel.pm line 824

[suparn777]

Hello Iguinn,

Thanks a lot for the answer.However there is no field called message and sample of errors is seen below:

[Sat Jun 2 10:5:46 2014] -e: No Pre-Fill Match. CID: 7c640e59b8e66be3c7061bd60d42c81b, EID: personalbandr, RID: 01_27_07, VAL: 0
host = abc source = /var/log/apache/error.log sourcetype = apache_error

6/2/14
5:05:46.000 PM

[Sat Jun 21 10:05:46 2014] -e: No Pre-Fill Match. CID: 7c640e59b8e66be3c7061bd60
host = abc source = /var/log/apache/error.log sourcetype = apache_error

Is there any other way in which we can create an alert?

Splunk version is 6.1.1

[lguinn2]

Version of Splunk?

How do you define "same type of error"?