I am running Ubuntu version 14 with Splunk version 6.2.2 and have noticed some strangeness with the Ulimits not being respected when the server is rebooted and Splunk is restarted by the OS.
The following edits were made to the /etc/security/limits.conf file:
splunk hard nofile 20240
splunk soft nofile 10240
The /etc/pam.d/sudo and /etc/pam.d/su files were also edited to add:
session required pam_limits.so
Upon starting Splunk manually (i.e. running splunk start from the CLI), the splunkd logs clearly show a limits setting of 20240. However, after a reboot of the server, these limits are not respected and the splunkd logs show an open file limit number of 4096.
Note - I ran the following after installing Splunk to ensure Splunk restarts after a reboot of the server.
splunk enable boot-start
It would appear that there is a bug/issue with Ubuntu and perhaps similar Linux OS's where the ulimits settings are not respected, resulting in the above characteristics.
Upon running the splunk enable boot-start command, a Splunk script is added to the /etc/init.d/ directory. A verified fix for this issue involves editing the Splunk script and hard-coding in the hard and soft ulimits to the Splunk script as follows:
Script Modification:
splunk_start() {
echo Starting Splunk...
ulimit -Hn 20240
ulimit -Sn 10240
"/opt/splunk/bin/splunk" start --no-prompt --answer-yes
RETVAL=$?
}
Note: The init.d Splunk script is created only AFTER you have installed Splunk and run the splunk enable boot-start command.
We ran into this same issue on RHEL 7. After some digging, we found that daemons loaded by systemd ignore anything in /etc/security/limits.conf AND limits.d/*
We wound up extending the service with a file in /etc/systemd/system that sets the limits.
Here are the steps we followed:
First, create directory /etc/systemd/system/splunk.service.d/ if it does not exist.
Then create a file in that directory called "filelimit.conf"
The contents of filelimit.conf should look like this (replacing the limit with whatever value you need):
[Service]
LimitNOFILE=65535
We found this better than trying to modify the init.d script on systems that utilize systemd for startup.
Thanks @brian.rampley ! Running RHEL 7.2 and indeed the /etc/security/limits.conf
was being ignored. My ulimit -a
value and what was showing up in splunk (from the health check) were different.
splunk.service.d
directorysystemctl daemon-reload
Did the trick!
It should be mentioned that this works (on CentOS 7) after a full host reboot. Just restarting splunk is insufficient.
By default most modern UNIX systems have pam_limits enabled, and /etc/security/limits.d/
Instead of editing /etc/security/limits.conf, create a file /etc/security/limits.d/99-splunk.com with the following
splunk hard nofile 20240
splunk soft nofile 10240
This just makes it easier to add/remove via any automation (puppet, chef..) or included in your own packages. This will also work if you for some reason chose to run anything splunk as 'root' too 😞 But that is an entirely different problem
(RBAC and facls) are your friend 😉
Typo - the last command should have been -
splunk enable boot-start -user splunk
I am running Splunk as the Splunk user, but this issue also appears to affect the root user for those running Splunk as root.
Hi,
As per your configuration for hardlimit and softlimit, you are setting ulimit for splunk user & you are running splunk as splunk user.
So can you please setup a bootscript with below command if you are running splunk as splunk user
$SPLUNK_HOME/bin/splunk enable boot-start -user splunk
I hope this solves your issue.
Thanks,
Harshil
Thank you sharing great answers
I was running it as a splunk user - just forgot to add that line. Even if you add this - it makes no difference. The issue even affects the root user (if you run splunk as root). The only solution I have found to date is the edit to the script.
http://docs.splunk.com/Documentation/Splunk/6.6.2/Admin/ConfigureSplunktostartatboottime
Check out this not in docs about updating the init.d for splunk user
@mcronkrite, Thanks a lot.
That's great to see that the doc has been finally updated and the problem taken in consideration, even if it took ages 😉
But never too late!
It would appear that there is a bug/issue with Ubuntu and perhaps similar Linux OS's where the ulimits settings are not respected, resulting in the above characteristics.
Upon running the splunk enable boot-start command, a Splunk script is added to the /etc/init.d/ directory. A verified fix for this issue involves editing the Splunk script and hard-coding in the hard and soft ulimits to the Splunk script as follows:
Script Modification:
splunk_start() {
echo Starting Splunk...
ulimit -Hn 20240
ulimit -Sn 10240
"/opt/splunk/bin/splunk" start --no-prompt --answer-yes
RETVAL=$?
}
Note: The init.d Splunk script is created only AFTER you have installed Splunk and run the splunk enable boot-start command.
thanks for this, fixed our issue by adding this to the splunk_start( ) function
Is this file overwritten every time you upgrade? We are also experiencing this issue and looking for the most practical solution.
Hi, Splunk does not overwrite the init.d script during upgrade.
It is generated once when you run the "enable boot-start" command during initial installation.