Is there a way I can be notified(via email) if splunkd stops when monitoring Splunkd services?

[Mansi24]

Hi,

My Splunkd service in production instance is automatically getting stopped. It has happened couple of times before as well. We don't get to know untill we or clients try logging in. Is there a way we can be notified if the services are getting stopped.

Please suggest.

[gcusello]

Hi Mansi24,
if you're speaking of splunkd of a Universal Forwarder it's easy.
If instead you're speaking of splunkd of a Search Head, an Indexer or another Splunk server, it's possible only if you have a redundant architecture: in other words, you can detect that an Indexer is down only if you have another Indexer up otherwise it isn't possible.

Anyway, to detect down servers you have to create a lookup with all the servers to monitor (e.g. perimeter.csv) in which there a field called host; then you have to run a search like this and associate it to an alert to schedule e.g. every five minutes:

index=_internal earliest=-5m latest=now
| eval host=lower(host)
| stats count BY host
| append [ | inputlookup perimeter.csv | eval count=0, host=lower(host) | fields count host ]
| stats sum(count) AS total BY host
| where total=0

You can use this search both for Universal Forwarders and/or Splunk servers, it depends by the perimeter you define.

Ciao.
Giuseppe

[Mansi24]

Thanks for the response gcusello!

Yes i am talking about my Search head here, forgot to mention that. Correct me if i am wrong, i need to create a lookup having all indexers and search head listed and try running this query. Would it work if my SH is down?

[gcusello]

Hi Mansi24,
If you have only one Search Haed this alert cannot run because alerts run on SHs.
If you have more than one Search Head, you can schedule this alert on all of them so you'll have your alert.

Ciao.
Giuseppe

[Mansi24]

just an update i am using a shell script on my linux server where SH is hosted, which checks the splunk service every 5 minutes and starts it if not running.Also sends me email when not running.