Deployment Architecture

Search head cluster member does not come up after exec restart cmd from master

immortalraghava
Path Finder

Hi,

I have a three search head SHC.

I see that one SHC member going for restart but never comes back up. This is the log line.

INFO SHCSlave - event=SHPSlave::handleHeartbeatDone master has instructed peer to restart

SHC has three members with Dynamic captain.

What could be going wrong.

Please help.

0 Karma

lycollicott
Motivator

So many things could be going wrong...so very many.

What happens when you logon to that search head and just do splunk start?

0 Karma

p_gurav
Champion

Hi immortalraghavan,
Can you please tell me output of "/splunk show shcluster-status" command?

0 Karma

mayurr98
Super Champion

http://answers.splunk.com/answers/82275/why-is-my-windows-cluster-peer-node-continually-restarting

Essentially, directory permissions on /slave-apps/ on the search peer had been lost (why?) and directory was set to read only. As per the link above, resetting the permissions allowed the Cluster Master to once again populate the directory with the required apps.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...