Knowledge Management

Why the KVstore process is being started as a root?

abhi04
Communicator

Splunk is not restarting because we are getting the error "kvstore port [8191] - port is already bound". After I check, I observed the process is starting as a root and so while restarting it assumes the port is being taken by another process. I killed the process and was able to start the splunk.

But I wanted to know the reason and the resolution to prevent this from happening in the future. I have checked and verified that the /var/lib/splunk/kvstore/mongo is owned by splunk. But some of the files such as "admin.0" "admin.ns" "config.0" and "config.ns" are owned as root and not splunk. Wanted to know what are those files and if these permissions should also be changed to splunk.
Also, the splunk.key have proper permission.

Labels (1)
0 Karma

codebuilder
SplunkTrust
SplunkTrust

Stop Splunk completely and verify all processes are down "ps -ef |grep -i splunk" e.g.
If any are still active, kill them off.

Modify the config at /opt/splunk/etc/splunk-launch.conf and ensure that SPLUNK_OS_USER is set to splunk.
SPLUNK_OS_USER=splunk

If you are using systemd, also verify the user is set correctly within the unit file in the [Service] stanza
User=splunk

Start Splunk back up and verify.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

codebuilder
SplunkTrust
SplunkTrust

Did this help resolve your issue? If so, please "accept" the answer so that others in the community may benefit.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

nickhills
Ultra Champion

This can happen if your instance was at some point started by root (perhaps by mistake)
All files in $SPLUNK_HOME should be owned by the user Splunk is running as (splunk)

If you have files inside $SPLUNK_HOME owned by root, you should probably run:
sudo chown -R splunk:splunk /opt/splunk - or the path of $SPLUNK_HOME

If my comment helps, please give it a thumbs up!
0 Karma

abhi04
Communicator

Hi @nickhillscpl,

The /opt/splunk is already owned as splunk.

I just wanted to know if there is a permanent fix for this. will the re-installation of splunk resolve this permanently?

0 Karma

nnimbe1
Path Finder

Can we delete old dated .ns files from $Splunk Directory$\Splunk\var\lib\splunk\kvstore\mongo folder to increase the SH drive space...whether it will have any impact on SH performance

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...