Deployment Architecture

Why Splunkd daemon not responding after distributing configuration bundle with master node?

scc00
Contributor

I have the following configuration: 1 Master Node, 1 Search Head, 2 Peers running version 6.1.1. RF 2, SF 1. When I update the indexes.conf within */master_apps/_cluster on the master node and then attempt to Distribute the Configuration Bundle to the peers, the master node disconnects and acts as though it is restarting. Any thoughts? Within the splunkd.log it shows the following: Note, I have rebooted the master node and ensured the peers were all online, the distribution bundle does not take.

Splunkd daemon is not responding: ('Error connecting to /services/cluster/master/control/default/apply: The read operation timed out',)

Splunkd logs

07-10-2014 13:00:27.296 -0400 WARN  CMMaster - event=removePeerBuckets peer=38E77442-7569-4EA1-A90F-0C76B0AF4D8F bid=main~5~043DCACA-C500-485A-9C52-84D2F2DE6C2D msg="Bucket is not on any other peer! Removing it."
07-10-2014 13:00:27.296 -0400 WARN  CMMaster - event=removePeerBuckets peer=38E77442-7569-4EA1-A90F-0C76B0AF4D8F bid=main~5~38E77442-7569-4EA1-A90F-0C76B0AF4D8F msg="Bucket is not on any other peer! Removing it."
07-10-2014 13:00:27.296 -0400 WARN  CMMaster - event=removePeerBuckets peer=38E77442-7569-4EA1-A90F-0C76B0AF4D8F bid=main~6~043DCACA-C500-485A-9C52-84D2F2DE6C2D msg="Bucket is not on any other peer! Removing it."
07-10-2014 13:00:27.296 -0400 WARN  CMMaster - event=removePeerBuckets peer=38E77442-7569-4EA1-A90F-0C76B0AF4D8F bid=main~6~38E77442-7569-4EA1-A90F-0C76B0AF4D8F msg="Bucket is not on any other peer! Removing it."
07-10-2014 13:00:27.296 -0400 WARN  CMMaster - event=removePeerBuckets peer=38E77442-7569-4EA1-A90F-0C76B0AF4D8F bid=main~7~043DCACA-C500-485A-9C52-84D2F2DE6C2D msg="Bucket is not on any other peer! Removing it."
07-10-2014 13:00:27.296 -0400 WARN  CMMaster - event=removePeerBuckets peer=38E77442-7569-4EA1-A90F-0C76B0AF4D8F bid=main~7~38E77442-7569-4EA1-A90F-0C76B0AF4D8F msg="Bucket is not on any other peer! Removing it."
07-10-2014 13:00:27.296 -0400 WARN  CMMaster - event=removePeerBuckets peer=38E77442-7569-4EA1-A90F-0C76B0AF4D8F bid=main~8~38E77442-7569-4EA1-A90F-0C76B0AF4D8F msg="Bucket is not on any other peer! Removing it."
07-10-2014 13:00:27.296 -0400 WARN  CMMaster - Apply bundle - all peers are down. Will not switch bundles.
0 Karma

ofrachon
Path Finder

Hello,

It looks like your Master Node does not "see" any of your peers.

It could be a configuration problem (i.e. secret key not properly shared) or a network problem.
Make sure the [clustering] stanza in the server.conf files on all members of the cluster (master node, search head, peers) are homogeneous.

Relevant stuff can be found on the Splunk Docs site : http://docs.splunk.com/Documentation/Splunk/6.1.2/Indexer/Clusterdeploymentoverview

Also, you should NOT put anything in $SPLUNK_HOME/etc/master-apps/_cluster as it is reserved for splunk stuff.

Instead you should put your apps in $SPLUNK_HOME/etc/master-apps/app1, $SPLUNK_HOME/etc/master-apps/app2 and so on.

Stuff regarding the Configuration bundle :
http://docs.splunk.com/Documentation/Splunk/6.1.2/Indexer/Updatepeerconfigurations

Regards.

Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...