How do you raise an alert on the condition that a search has been modified and include only new events

[varunghai]

Hi,

I have written a search query which generates alert every 1hr and gets the information of some jobs depending on their time of last execution. Now the issue I am facing is that when my 1st alert is generated and it has the info of say jobA which has not run on its expected time, in the second alert the jobA will still be there if it still hasn't run yet.

So I want to modify my search query to generate alert for 1 job only once and not include that particular job in every alert again and again if it is being returned based on the same criteria.

Search QUery:
index=tomcat [Job] cronjob earliest=@d| sort CronJobName | search Action="Starting" | eventstats latest(_time) as _time by CronJobName | eval Time_Diff_Min=(now()-_time)/60 | eval _time=strftime(_time,"%d-%m-%Y %H:%M")|eval Time_Diff_Min=round(Time_Diff_Min)|dedup Action CronJobName | join CronJobName type=inner[|inputlookup CronJobLookup.csv] | table Job_Frequency_min Action _time Time_Diff_Min CronJobName Expected_Start_Time | eval epoch_a=now()| eval CurrentDate=strftime(now(),"%d-%m-%Y")|eval epoch_b=strptime(CurrentDate." ".Expected_Start_Time,"%d-%m-%Y %H:%M"),ExpectedTime=strftime(epoch_b,"%d-%m-%Y %H:%M"), CurrentTime=strftime(epoch_a,"%d-%m-%Y %H:%M") |eval Time_Diff_Hour=Time_Diff_Min/60|eval Time_Diff_Hour=round(Time_Diff_Hour,1)|eval Job_Freq_Num=tonumber(Job_Frequency_min)|eval tt=if(Job_Frequency_min!="Once",Job_Freq_Num+60,0)|eval CronJobAlert=case(Job_Frequency_min="Once" AND (Time_Diff_Hour>24.5) ,1,Job_Frequency_min!="Once" AND (Job_Freq_Num+10

[DalJeanis · Accepted solution]

It's pretty simple. Don't worry about the query at all.

In the alert specifications, you need to "throttle" the alert for your chosen length of time, based upon the key fields.

http://docs.splunk.com/Documentation/Splunk/6.3.1/Alert/ThrottleAlerts

https://answers.splunk.com/answers/491058/how-to-edit-my-search-to-alert-once-per-result-for.html

[rithwik572]

Hi @DalJeanis,
I am trying to generate alert when a jenkins slave memory get's full. The search string I am using is index=app_devops AND "No space left on device" source="/opt/artifacts/jenkins/log/jenkins*"
Event's shown are as follows:
Caused by: java.io.IOException: remote file operation failed: /opt/jenkins/7cb92e15/tools/hudson.model.JDK/IBM_1.7_JDK at hudson.remoting.Channel@1901245e:Linux-build-slave-12: java.io.IOException: No space left on device

I would like to print the slave name ( Linux-build-slave-12) on the email body of alert generated in real time as we have 30 slaves running in our environment . current alert doesn't show the slave name, we need to look for full output to determine the slave name. Please let me know how to configure it,
Thanks in advance

[varunghai]

I would prefer to have that logic to understand what is missing or what needs to be modified rather than changing the whole query