Timed out waiting for splunkd to shutdown

[kenison]

I can't restart splunk. Neither CLI or web manager works. It just times out waiting for splunkd to shut down. Thing is, if I kill -9 everything splunk, it still thinks it needs to stop splunkd and splunkweb. And If I kill everything and try to start, it thinks splunkd is still running. This is really frustrating.

[jmccallhbo]

I am having a similar issue. We are starting Splunk with a golang script that runs the command /opt/splunk/bin/splunk start. This leaves behind a zombie splunkd process but otherwise starts fine. Running splunk stop will time out and it prints that it timed out to the console. Interestingly splunk does stop in less than 2 secs and the splunk.pid file is removed. What is splunk stop waiting for???

[Simeon]

Sounds like splunk is hung trying to execute or close an activity. When running forwarders, Splunk will close these connections to the indexer gracefully. In some cases, Splunk might be still writing to disk and is waiting for that activity to complete.

The best thing to do is look at the $SPLUNK_HOME/var/log/splunkd.log file and check for any errors during shutdown. Additionaly, splunk creates *.pid files that map if Splunk is still running or not. If those pid files still exist, Splunk will not start. In that case, you could manually remove them.