Why pushing bundled apps hangs the splunkd service with error "No appservers running"?

[jsmith39]

I'm trying to build a clustered index in a lab environment between 3 servers (1 master, 2 slaves). The indexer clustering appears to have worked correctly. In the distributed environment/indexer clustering dashboard, both slave indexers are showing as peers.

I placed 3 applications in the master-apps directory and attempted force replication through the CLI but was given an error:

“/root/.splunk”:permission denied

Out of curiosity, I then attempted to push the apps through UI (edit, distribute configuration bundle), and this was successful, but I was then unable to access either slave indexer through port 8000. I get a "No appservers running, server had an unexpected error" message. I'm able to ssh into the servers, but when I stopped splunk (../splunk stop) and then restarted it (/etc/init.d/splunk start), the service took several minutes to start and the end result was the "No appservers running" message mentioned above.

[jsmith39 · Accepted solution]

To follow up, this issue appears to be related to the splunk_app_for_nix.
On a rebuild of my master and 2 peers I took the following actions:

1) Connected the 3 servers as an indexer cluster
2) Created a custom app to push new indexes (that would be created by the apps I intend to upload) with the repFactor = auto switch added to each stanza.
3) Pushed the app bundle -success
4) Uploaded the Splunk_TA_windows app, and pushed it -success
5) Uploaded the TA-SOS app, and pushed it -success
6) Uploaded the splunk_app_for_nix, and pushed it -Fail, system goes into a loop and I get the "No App Server" message when trying to access the UI.
7) Turn splunkd off on all servers, delete the splunk_app_for_nix, restart the splunkd service and everything goes back to normal.

[napomokoetle]

I have similar issue on Splunk running on Windows. Don't know which log to look into to troubleshoot the problem. Look forward to leaning how to resolve this "No appservers running" issue.