I am looking into writing something that will check the health of my indexer and search head clusters. This will eventually be put into an enterprise monitoring tool and alerts sent out if there is an "issue". I know you can run splunk show cluster-status
etc... We are running this all on Linux and I am thinking that someone may have a shell script that can be used to verify health of this environment.
Any suggestions?
I use the bash alias command called splunkstatus using the following command and script.
alias splunkstatus=' "ssh <search_head> "sudo -u <user_running_splunk> /opt/splunk/bin/splunk show shcluster-status -auth <user>:<password>" | ~/statusparse.sh'
SCRIPT:
#!/bin/bash
count=1;
output="$(/usr/bin/awk '{print $3}')"
while read line;
do
if [[ $line =~ ^https://<prefix>*. ]]; then
echo "Search Captian: $line";
fi;
if [[ $line =~ ^<search_prefix>*. ]]; then
search="$line";
fi;
if [[ $line =~ ^Down|Up|Restarting ]]; then
search="$count $search $line";
echo "$search";
((count++))
fi;
done <<< "$output"
You'll need to mod it a bit.
For a very gross view, you can use $SPLUNK_HOME/bin/splunk status
.
Have you seen Splunk Health Overview app?? This does provide health of Splunk servers (whichever are added as search peer) using REST API endpoints using search.
https://splunkbase.splunk.com/app/1919/