Alerting

Alerting for splunkd

Ant1D
Motivator

Hey,

I have seen the following messages in the past when using/trying to access the Splunk UI:

Splunkd daemon is not responding...

The splunkd daemon cannot be reached by splunkweb

I would like to set up some alerts to inform me when the above events happen. How can this be done?
Does anyone have a ready-made search query for this? Is it simply the case of building search queries that look for such phrases in the internal logs (splunkd.log)?

Thanks in advance for your help.

mhassan
Path Finder

Here is a simple script that test for port 8089 connectivity. It will send email alert (and restart the process if not connection):

/export/home/mhassan/scripts $ cat splunkd-watchdog.sh

!/bin/bash

splunkd-watchdog.sh

Monitor Splunk port 8089. Restart process if no answer

-Mohamad Hassan 2/2/215

timestamp=date
USER=whoami
WHO=who -u
host=uname -n

LOGFILE="/opt/splunk/var/log/splunk-watchdog.log"
PROC="splunkd"
PORT="8089"
ADMINS="user1@domain.com user2@domain.com"
TEST=nc -z -w5 $HOSTNAME $PORT;echo $? < /dev/null

Debugging

let TEST=0

echo $TEST

if [ $TEST -eq 1 ] ;then

    /opt/splunk/bin/splunk restart
    RESULT=`ps ax | grep -i "$PROC" | grep -v grep|grep -v watchdog`

    echo  "$timestamp $HOSTNAME: splunk-watchdog ALERT! $PROC process restarted" >> $LOGFILE

    echo -e "Splunk watchdog ALERT! Splunk port $PORT is not answering.\n\n Hostname:    $HOSTNAME\n Time:               $timestamp\n Notifying:      $ADMINS\n ACTION:       Restarting ($PROC)!\n\nCurrrent logged-in users:\n$WHO\n\nCurrent splunkd status(after restart):\n$RESULT" |mail -s "$PROC watchdog ALERT! [$HOSTNAME]"  $ADMINS

    echo "$PROC restarted!"
    echo "Email sent to: $ADMINS"

fi

0 Karma

wrangler2x
Motivator

I don't know what sort of output you'd like with this search but this would do the job by providing a count that you can alert on (count > 0).

index=_internal source="/opt/splunk/var/log/splunk/web_service.log" log_level="error" sourcetype="splunk_web_service"  "Splunkd daemon is not responding" | stats earliest(_time) as FirstOccurance, latest(_time) as LastOccurance count |eval FirstOccurance=strftime(FirstOccurance,"%m/%d/%Y %H:%M:%S")|eval LastOccurance=strftime(LastOccurance,"%m/%d/%Y  %H:%M:%S")

If you don't have an occurrence of this in your current splunkd.log you can change the "source" part to end with a '*' and it will look in all the rotated-out logs as well as the current. Then just set the time window to go back to when you had one of these.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...