How to edit my search to find if a service is down, then trigger an alert only if the status is still down 1 minute later?

[splunker9999]

Hi ,

We have search that runs for every minute, and if in case it found any Service is down, it triggers an alert. However, we are thinking to enhance the search in a way that search should run for every 1 min interval, and if it finds a service down, it should not trigger an alert. Instead, it should wait for one more minute and if it still gets the status as "Service Down", then it should trigger.

Below is our basic search:

index =index1 sourcetype=WMI Caption="*" host=WGP 
| stats latest(State) AS State by _time host Name
| rename Name as Service
| search State=Stopped
| eval currentTime=now() 

[sundareshr · Accepted solution]

Try this. Run this every two mins and trigger if count >= 2

 index =index1 sourcetype=WMI Caption="*" host=WGP earliest=-2m@m State="Stopped"
| bin span=2m _time
| stats stats count by _time host Name
| rename Name as Service
| eval currentTime=now() 

[Raschko]

The bin span should be 1min correct?

[splunker9999]

Hi,

Is bin span to be 1m?

Thanks

[sundareshr]

You want it to be 2m. If you have this scenario twice in quick succession, you want the alert, right?

[splunker9999]

Yes, but if we give this condition and ran this search ,and found now results which is correct, but when scheduled this alert some how we are getting alerts with out satisfying conditions?

Will this condition will work, I have just added where count>=2

 index =index1 sourcetype=WMI Caption="*" host=WGP earliest=-2m@m State="Stopped"
 | bin span=2m _time
 | stats stats count by _time host Name|where count>=2
 | rename Name as Service
 | eval currentTime=now()

This is because, we have script running ,which will raise automatic service now ticket if it finds any results, since we have specified trigger condition count>=2 ,we are not getting alert but it is raising automatic Service now ticket ,so I thought of changing this condition in Query itself?

[sundareshr]

Yes, it should work