Unable to Control SPLUNKD

[pfabrizi]

I have been having space issue on one of my indexes running SPLUNK 6.5.1. The box appears to crash from time to time. Typically I have to restart SPLUNK, but today I am unable to control SPLUNKD. I get this when I issue a splunk command:

Stopping splunk helpers...
couldn't send SIGTERM to pid 2972: Operation not permitted
Couldn't send SIGTERM to some splunk helpers. [FAILED]
Error: Unable to stop splunk helpers.

What process\task is running this, I am guessing I can kill this?

Thanks!

[vinkumar_splunk]

This could be because of stale PID files which reside under SPLUNK_HOME/var/run/splunk that causes splunk not to start.

Step 1:

rm $SPLUNK_HOME/var/run/splunk/splunkd.pid

Step 2:

$SPLUNK_HOME/bin/splunk status

Step 3:

$SPLUNK_HOME/bin/splunk start

[cain1288]

This resolved my issue, thank you.

Not sure what the root cause was, and have not seen this before.

Steps to produce issue were:

1) yum update on RHEL 7 server

2) initiate reboot command (perhaps something here cause the issue? Splunkd did not completely stop?)

3) attempt to start splunk resulted in error: couldn't send sigterm to pid 332 operation not permitted

4) Resolution was to delete the /var/run/splunk/splunkd.pid as mentioned above, splunkd started on the next attempt

[mcazacu]

Fixed it for me. Thank you @vinkumar_splunk !!! 🙂

[nickhills]

Operation not permitted

suggests that you do not have permission to terminate the process.

Assuming your on linux, run:

ps -ef|grep splunkd

look to see which users are running the process named splunkd -p 8089 start (it will likely be root or splunk, and there may be two)
If you have one for each user, this is fine, but if they are both root, that will be your issue.

This can occur if someone forgets to run splunk as the splunk user (ie root restarts it) instead of sudoing to splunk.

[pfabrizi]

I am on Linux, I ran 'p's looking for splunkd and nothing running. I believe they did restart the server last night. I just checked and my root volume is out of space (at 99% used). Can that be the issue?

[nickhills]

Ahh, in that case it sound like Splunk is not running, but it was not stopped gracefully (perhaps because it ran out of disk) and thus it’s not cleared it’s pid.

I guess either that pid is no longer running, or (unluckily) something else now has it. Hence the ‘not permitted’

[masonmorales]

Yes, you need to free up some disk space for Splunk to run properly.