Hi Team,
I got a scenario as below:
index=* host=A or host=B Type=Info "Service down"
In this i want the following thing needs to be achieved.
1.) If host A or host B service is down then it needs to send a warning alert in Hipchat.
2.)Next condition is that if both Host A as well as Host B service is down then it needs to send a critical alert to Hipchat.
Can i know whether it can be achieved in a single query itself?
So if not, then if we need to create two alerts separately then anyhow in Hipchat it will trigger two alerts but our requirement is only one. Either any one of the server is down then go as warning if both is down it needs to go as critical but it needs to be only one alert.
Kindly note we have already configured Hipchat in our environment and integrated to Splunk.
So i just need help in the search query. And also let me know whether it can be achieved,
This query you can used:
index=* host=A or host=B Type=Info "Service down" | stats dc(host) AS hostcount| eval abc=if(hostcount=1, "warning", "critical")
Then you can you pass token in email subject:
https://docs.splunk.com/Documentation/Splunk/7.0.2/Alert/EmailNotificationTokens
Thanks for your response @p_gaurav. But the required condition is not met.
For example currently there is no service down in any of the hosts but still i can see the result as value "0" and it shows as critical when i searched the logs for last 15 minutes.
So we require the output to be as :
If the service is down in any of the host then it needs to be as Warning. And if both of the service down in both the hosts then it should be critical.
This is our requirement, but here in the solution you provided if there is a service down in any of the server (host=1) condition then it throws warning that's fine. but if none of the services were down in any of the servers then its throwing error as " 0 and critical" so kindly check and update.
index=* host=A or host=B Type=Info "Service down" | stats dc(host) AS hostcount| eval abc=case(hostcount=1, "warning", hostcount>1, "critical") | stats values(abc) AS abc
Then in alert condition put resultcount > 0
Thanks for response.
But I want to know how to set the severity as warning for one alert down and critical if both goes down.
My intention is that when it reaches HipChat room if any one server is down it needs to go as warning in Hipchat and if both down then it needs to go as critical in Hipchat but only one alert.
We can configure as two alerts also but it needs to go as only one alert in Hipchat how to achieve it.
Can anyone help on this request
This may help you:
https://answers.splunk.com/answers/32385/alert-script-and-severity.html
And if you want to edit subject line then you can use $result.abc$ .