Splunk Search

Search Query Help

anandhalagarasa
Path Finder

Hi Team,

I got a scenario as below:
index=* host=A or host=B Type=Info "Service down"

In this i want the following thing needs to be achieved.
1.) If host A or host B service is down then it needs to send a warning alert in Hipchat.
2.)Next condition is that if both Host A as well as Host B service is down then it needs to send a critical alert to Hipchat.

Can i know whether it can be achieved in a single query itself?

So if not, then if we need to create two alerts separately then anyhow in Hipchat it will trigger two alerts but our requirement is only one. Either any one of the server is down then go as warning if both is down it needs to go as critical but it needs to be only one alert.

Kindly note we have already configured Hipchat in our environment and integrated to Splunk.

So i just need help in the search query. And also let me know whether it can be achieved,

Tags (1)
0 Karma

p_gurav
Champion

This query you can used:

index=* host=A or host=B Type=Info "Service down" | stats dc(host) AS hostcount| eval abc=if(hostcount=1, "warning", "critical")

Then you can you pass token in email subject:
https://docs.splunk.com/Documentation/Splunk/7.0.2/Alert/EmailNotificationTokens

0 Karma

anandhalagarasa
Path Finder

Thanks for your response @p_gaurav. But the required condition is not met.

For example currently there is no service down in any of the hosts but still i can see the result as value "0" and it shows as critical when i searched the logs for last 15 minutes.

So we require the output to be as :

If the service is down in any of the host then it needs to be as Warning. And if both of the service down in both the hosts then it should be critical.

This is our requirement, but here in the solution you provided if there is a service down in any of the server (host=1) condition then it throws warning that's fine. but if none of the services were down in any of the servers then its throwing error as " 0 and critical" so kindly check and update.

0 Karma

p_gurav
Champion
     index=* host=A or host=B Type=Info "Service down" | stats dc(host) AS hostcount| eval abc=case(hostcount=1, "warning", hostcount>1, "critical") | stats values(abc) AS abc

Then in alert condition put resultcount > 0

0 Karma

anandhalagarasa
Path Finder

Thanks for response.

But I want to know how to set the severity as warning for one alert down and critical if both goes down.

My intention is that when it reaches HipChat room if any one server is down it needs to go as warning in Hipchat and if both down then it needs to go as critical in Hipchat but only one alert.

We can configure as two alerts also but it needs to go as only one alert in Hipchat how to achieve it.

0 Karma

anandhalagarasa
Path Finder

Can anyone help on this request

0 Karma

p_gurav
Champion

This may help you:
https://answers.splunk.com/answers/32385/alert-script-and-severity.html

And if you want to edit subject line then you can use $result.abc$ .

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...