I'm getting this error when I try to push an app to my search head cluster:
Error when getting master uri from target to do a rolling-restart Service Unavailable
I've used the command:
sudo ./splunk apply shcluster-bundle -target https://[SOME SEARCH HEAD]:8089 -auth admin:'[THE PASSWORD]'
It stopped working since we've fixed the IP addresses of the Splunk nodes. Anyone an idea how to fix/debug this?
You need to correct anything wrong with your SHC first.
Run this command from your search head(s) and let us know the output:
splunk show shcluster-status -auth <username>:<password>
Also it is best practice to create your own splunk user & group, and run splunk as that user.
#If splunk is running as sudo or root, stop it first.
groupadd splunk
useradd splunk -g splunk
chown -Rf splunk. /opt/splunk #or wherever you have splunk installed
You'll probably have to fix/rebuild the SHC using some of these commands from each search head:
splunk init shcluster-config -auth <username>:<password> -mgmt_uri <URI>:<management_port> -replication_port <replication_port> -replication_factor <n> -conf_deploy_fetch_url <URL>:<management_port> -secret <security_key>
splunk restart
http://docs.splunk.com/Documentation/Splunk/6.2.0/DistSearch/ViewSHCstatus
http://docs.splunk.com/Documentation/Splunk/6.2.0/DistSearch/SHCdeploymentoverview
You need to correct anything wrong with your SHC first.
Run this command from your search head(s) and let us know the output:
splunk show shcluster-status -auth <username>:<password>
Also it is best practice to create your own splunk user & group, and run splunk as that user.
#If splunk is running as sudo or root, stop it first.
groupadd splunk
useradd splunk -g splunk
chown -Rf splunk. /opt/splunk #or wherever you have splunk installed
You'll probably have to fix/rebuild the SHC using some of these commands from each search head:
splunk init shcluster-config -auth <username>:<password> -mgmt_uri <URI>:<management_port> -replication_port <replication_port> -replication_factor <n> -conf_deploy_fetch_url <URL>:<management_port> -secret <security_key>
splunk restart
http://docs.splunk.com/Documentation/Splunk/6.2.0/DistSearch/ViewSHCstatus
http://docs.splunk.com/Documentation/Splunk/6.2.0/DistSearch/SHCdeploymentoverview
We've rebuild the SHC (splunk init shcluster-config ... ) and started the election for captain again. Then things started working again. Thanks guys!
I don't know if this causes your problem, but generally, I'd advise against sudoing splunk. It causes all kind of undesired behavior with folder/file ownership and permissions if some splunk actions are performed as a different user.