Knowledge Management

Why the KVstore process is being started as a root?

abhi04
Communicator

Splunk is not restarting because we are getting the error "kvstore port [8191] - port is already bound". After I check, I observed the process is starting as a root and so while restarting it assumes the port is being taken by another process. I killed the process and was able to start the splunk.

But I wanted to know the reason and the resolution to prevent this from happening in the future. I have checked and verified that the /var/lib/splunk/kvstore/mongo is owned by splunk. But some of the files such as "admin.0" "admin.ns" "config.0" and "config.ns" are owned as root and not splunk. Wanted to know what are those files and if these permissions should also be changed to splunk.
Also, the splunk.key have proper permission.

Labels (1)
0 Karma

codebuilder
Influencer

Stop Splunk completely and verify all processes are down "ps -ef |grep -i splunk" e.g.
If any are still active, kill them off.

Modify the config at /opt/splunk/etc/splunk-launch.conf and ensure that SPLUNK_OS_USER is set to splunk.
SPLUNK_OS_USER=splunk

If you are using systemd, also verify the user is set correctly within the unit file in the [Service] stanza
User=splunk

Start Splunk back up and verify.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

codebuilder
Influencer

Did this help resolve your issue? If so, please "accept" the answer so that others in the community may benefit.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

nickhills
Ultra Champion

This can happen if your instance was at some point started by root (perhaps by mistake)
All files in $SPLUNK_HOME should be owned by the user Splunk is running as (splunk)

If you have files inside $SPLUNK_HOME owned by root, you should probably run:
sudo chown -R splunk:splunk /opt/splunk - or the path of $SPLUNK_HOME

If my comment helps, please give it a thumbs up!
0 Karma

abhi04
Communicator

Hi @nickhillscpl,

The /opt/splunk is already owned as splunk.

I just wanted to know if there is a permanent fix for this. will the re-installation of splunk resolve this permanently?

0 Karma

nnimbe1
Path Finder

Can we delete old dated .ns files from $Splunk Directory$\Splunk\var\lib\splunk\kvstore\mongo folder to increase the SH drive space...whether it will have any impact on SH performance

0 Karma
Get Updates on the Splunk Community!

Get the T-shirt to Prove You Survived Splunk University Bootcamp

As if Splunk University, in Las Vegas, in-person, with three days of bootcamps and labs weren’t enough, now ...

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Wondering How to Build Resiliency in the Cloud?

IT leaders are choosing Splunk Cloud as an ideal cloud transformation platform to drive business resilience,  ...