Installation

Splunk 7.0.1 Init script is not configured to run at boot

rfitch
Path Finder

Just upgraded a heavy forwarder to 7.0.1 and I'm not able to enable boot start.

I've tried:
/opt/splunk/bin/splunk enable boot-start
/opt/splunk/bin/splunk enable boot-start -user root

neither seem to work.

[root@dvplagsplnk06 opt]# /opt/splunk/bin/splunk enable boot-start
Note: Forwarding request to 'systemctl enable splunk.service'.
Created symlink from /etc/systemd/system/multi-user.target.wants/splunk.service to /usr/lib/systemd/system/splunk.service.
Note: Forwarding request to 'systemctl disable splunk.service'.
Removed symlink /etc/systemd/system/multi-user.target.wants/splunk.service.
Init script installed at /etc/init.d/splunk.
Init script is not configured to run at boot.

I confirmed with display boot-start and even rebooted the server and the Splunk service doesn't start.

I also tried manually running

systemctl enable splunk.service

with no luck.

I'm on Rhel 7.3
uname -a
Linux dvplagsplnk06 3.10.0-514.26.2.el7.x86_64 #1 SMP Fri Jun 30 05:26:04 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.3 (Maipo)

Tags (1)
0 Karma
1 Solution

rfitch
Path Finder

Follow up from my very excellent call with support

Looks like Chef was laying down a unit file for Splunk that not only conflicted with the Splunk RPM version, but was also incomplete. The fix was to take the unit file from https://answers.splunk.com/answers/590209/rhelcentos-7-systemd-not-honoring-ulimits.html and in my case use it to replace

 /usr/lib/systemd/system/splunk.service 

then run
systemctl daemon-reload
systemctl enable splunk.service

View solution in original post

bandit
Motivator

Summary of the issue:
Splunk 6.0.0 - Splunk 7.2.1 defaults to using init.d when enabling boot start
Splunk 7.2.2 - Splunk 7.2.9 defaults to using systemd when enabling boot start
Splunk 7.3.0 - Splunk 8.x defaults to using init.d when enabling boot start

systemd defaults to prompting for root credentials upon stop/start/restart of Splunk

Here is a simple fix if you have encountered this issue and prefer to use the traditional init.d scripts vs systemd.

Splunk Enterprise/Heavy Forwarder example (note: replace the splunk user below with the account you run splunk as):

sudo /opt/splunk/bin/splunk disable boot-start
sudo /opt/splunk/bin/splunk enable boot-start -user splunk -systemd-managed 0

Splunk Universal Forwarder example (note: replace the splunk user below with the account you run splunk as):

sudo /opt/splunkforwarder/bin/splunk disable boot-start
sudo /opt/splunkforwarder/bin/splunk enable boot-start -user splunk -systemd-managed 0

rfitch
Path Finder

Follow up from my very excellent call with support

Looks like Chef was laying down a unit file for Splunk that not only conflicted with the Splunk RPM version, but was also incomplete. The fix was to take the unit file from https://answers.splunk.com/answers/590209/rhelcentos-7-systemd-not-honoring-ulimits.html and in my case use it to replace

 /usr/lib/systemd/system/splunk.service 

then run
systemctl daemon-reload
systemctl enable splunk.service

sloshburch
Splunk Employee
Splunk Employee

Wow. I would have never guessed! Thanks for sharing the solution AND accepting it. This will certainly be informative for others!

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Sanity check question: Would you disable boot-start and then try to enable again?

0 Karma

rfitch
Path Finder

No dice:

[root@nvplagsplnk08 ~]# /opt/splunk/bin/splunk display boot-start 
Init script is not installed (checked: /etc/init.d/splunk).
Init script is not configured to run at boot.
[root@nvplagsplnk08 ~]# /opt/splunk/bin/splunk enable boot-start 
Note: Forwarding request to 'systemctl enable splunk.service'.
Created symlink from /etc/systemd/system/multi-user.target.wants/splunk.service to /usr/lib/systemd/system/splunk.service.
Note: Forwarding request to 'systemctl disable splunk.service'.
Removed symlink /etc/systemd/system/multi-user.target.wants/splunk.service.
Init script installed at /etc/init.d/splunk.
Init script is not configured to run at boot.
0 Karma

sloshburch
Splunk Employee
Splunk Employee

I notice you ran display, not disable. Good call on opening a case with support in parallel because as you can surmise, you might not be hitting something the rest of the community has run into.

0 Karma

marycordova
SplunkTrust
SplunkTrust

(ps (pre-script) hope this helps, apologies if I made any assumptions about your technical acumen!)

I don't believe in letting vendors off the hook...but set a boot cron job for now so you're not screwed until this gets fixed:

@reboot /etc/init.d/splunk start (or some such...you may have to prepend that with /bin/bash then the /etc...)

Of course you shouldn't run Splunk as root...but since Splunk elects not to provide a clear mechanism by which to do this, configure your system, still actually monitor logs, and bind to low numbered ports, I'll spare you the lecture.

If you get to the point where you're not running as root you can play around with the options in both the Splunk command to configure boot-start as a specific user or modify the cronjob to run init script as a specific user, example something I have logging to Splunk: @reboot sudo -u username '/opt/something/bin/someotherthing'

Also, I don't know if this was ever fixed but there was a bug where Splunk would ignore the ulimits on a system, so I added them to my init script directly *in both the start and restart functions*:

splunk_start() {
echo Starting Splunk...
ulimit -Hn 65536
ulimit -Sn 32768
"/opt/splunkforwarder/bin/splunk" start --no-prompt --answer-yes

...

splunk_restart() {
echo Restarting Splunk...
ulimit -Hn 65536
ulimit -Sn 32768
"/opt/splunkforwarder/bin/splunk" restart

@marycordova
0 Karma

sloshburch
Splunk Employee
Splunk Employee
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...