Alerting

How to build an alert that will trigger any server issue and do an automatic restart of the server through splunk alert without human intervention?

Prakash493
Communicator

Hi , I am looking for to automate jobs for splunk , i want to build an alert that will trigger if any server has issue as well as do a automatic restart of the server through the splunk alert without human intervention , How can i do it ?

0 Karma

woodcock
Esteemed Legend

Splunk recently bought Phantom so if you are looking for an add-on solution, I would start there because it will obviously be heavily supported and probably fully integrated at some point.

0 Karma

sduff_splunk
Splunk Employee
Splunk Employee

You can look at creating an alert action script, as documented at https://docs.splunk.com/Documentation/Splunk/7.2.5/AdvancedDev/ModAlertsIntro

You will need to write a Splunk query that identifies the hosts which are not responsive. At the end of the search, you want a list of servers. Save this search as an alert, then assign an alert action to it.

Your alert action script will need to read a CSV file which contains your servers, then decide what you want to do with them. I suggest using something like ssh with pre-shared certificates, then doing ssh root@remote-server.com /sbin/reboot . Just do that for each host that is listed in the CSV.

Splunk also supported very basic alert action scripts, which I summarised at https://simonduff.net/splunk_alert_script/ . This has been deprecated, but still works.

Alternatively, you can also look at Splunk Phantom. That has many more features, running advanced playbooks, etc... which is probably overkill for what you require.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...