Why is an updated ulimit for a Splunk user account on a forwarder not reflected in a Splunk search?

[saranya_fmr]

Hi All,

I updated the ulimit settings for a Splunk user account on a forwarder from 8192 to 10240.
I checked in the host(splnkdev01) using the ulimit command like below and it gives the updated value:
[splunkd1@splnkdev01 bin]$ ulimit -n
10240
However ,when I run the search below, it shows the older value for a particular sourcetype.

index=_internal host = splnkdev01* source=*splunkd.log ulimit

12/1/16
4:22:13.470 PM

12-01-2016 16:22:13.470 -0500 INFO ulimit - Limit: open files: 8192 files
host = splnkdev01 source = /opt/common/apps/vendor/splunk/forwarder/var/log/splunk/splunkd.log sourcetype = is_bs

Shouldn't it reflect the updated value 10240 for all? Why is it showing 8192 for a particular sourcetype?

Also I have updated max_fd to 1024in limits.conf.

[koshyk]

can u run a btool on the UF to see if the limits.conf is being overridden by any other limits.conf from another app?

Try changing all of the below
- the file size (ulimit -f)
- the number of open files (ulimit -n),
- number of file descriptors the number of user processes (ulimit -u)

Also did you do it temporarily? to do it permanently do it in /etc/security/limits.conf
set both the "hard" and "soft" limit too has to be set

Check your system wide max ulimit is higher. (/etc/sysctl.conf fs.file-max)

[saranya_fmr]

Im not able to attach images, so pasting the search/query results and command results:
1) Query -- index=_internal host = ABC* source=*splunkd.log ulimit
12/8/16
2:45:58.242 PM

12-08-2016 14:45:58.242 -0500 INFO ulimit - Limit: cpu time: unlimited
host = ABC source = /opt/common/apps/vendor/splunk/forwarder/var/log/splunk/splunkd.log sourcetype = ias_bass
12/8/16
2:45:58.242 PM

12-08-2016 14:45:58.242 -0500 INFO ulimit - Limit: user processes: 2060308 processes
host = ABC source = /opt/common/apps/vendor/splunk/forwarder/var/log/splunk/splunkd.log sourcetype = ias_bass
12/8/16
2:45:58.242 PM

12-08-2016 14:45:58.242 -0500 INFO ulimit - Limit: open files: 8192 files
host = ABC source = /opt/common/apps/vendor/splunk/forwarder/var/log/splunk/splunkd.log sourcetype = ias_bass
12/8/16
2:45:58.242 PM

12-08-2016 14:45:58.242 -0500 INFO ulimit - Limit: data file size: unlimited
host = ABC source = /opt/common/apps/vendor/splunk/forwarder/var/log/splunk/splunkd.log sourcetype = ias_bass
12/8/16
2:45:58.242 PM

12-08-2016 14:45:58.242 -0500 WARN ulimit - Core file generation disabled
host = ABC source = /opt/common/apps/vendor/splunk/forwarder/var/log/splunk/splunkd.log sourcetype = ias_bass
12/8/16
2:45:58.242 PM

12-08-2016 14:45:58.242 -0500 INFO ulimit - Limit: core file size: 0 bytes [hard maximum: unlimited]
host = ABC source = /opt/common/apps/vendor/splunk/forwarder/var/log/splunk/splunkd.log sourcetype = ias_bass
12/8/16
2:45:58.242 PM

12-08-2016 14:45:58.242 -0500 INFO ulimit - Limit: stack size: 10485760 bytes [hard maximum: unlimited]
host = ABC source = /opt/common/apps/vendor/splunk/forwarder/var/log/splunk/splunkd.log sourcetype = ias_bass
12/8/16
2:45:58.242 PM

12-08-2016 14:45:58.242 -0500 INFO ulimit - Limit: resident memory size: unlimited
host = ABC source = /opt/common/apps/vendor/splunk/forwarder/var/log/splunk/splunkd.log sourcetype = ias_bass
12/8/16
2:45:58.242 PM

12-08-2016 14:45:58.242 -0500 INFO ulimit - Limit: data segment size: unlimited
host = ABC source = /opt/common/apps/vendor/splunk/forwarder/var/log/splunk/splunkd.log sourcetype = ias_bass
12/8/16
2:45:58.242 PM

12-08-2016 14:45:58.242 -0500 INFO ulimit - Limit: virtual address space size: unlimited
host = ABC source = /opt/common/apps/vendor/splunk/forwarder/var/log/splunk/splunkd.log sourcetype = ias_bass

2)

I did a permanent change in etc/security/limits.conf.

Plesae see below for a forwarder as a splunk user

sh-3.2$ ulimit -n
10240
sh-3.2$
sh-3.2$
sh-3.2$ cat /proc/sys/fs/file-max
131072
sh-3.2$ ulimit -n
10240
sh-3.2$ ulimit -f
unlimited
sh-3.2$ ulimit -u
2060308
sh-3.2$ whoami
splunkuser
sh-3.2$ exit
exit
bash-3.2$
bash-3.2$
bash-3.2$ ulimit -n
8192

[saranya_fmr]

Hi @SloshBurch ,

That's my doubt. I don't understand why it's showing a particular sourcetype.

Hi @ddrillic ,

Exactly..!! Seems to be a mismatch..

[sloshburch]

@saranya_fmr - Please check with Erin/Jim why the sourcetype is is_bs and not splunkd. Also, please add sceenshots instead of pastes of text if possible. That will confirm if some other details are missing that are helpful for this.
Lastly, please confirm what user splunk is running as on the splnkdev01 with:
hostname && ps -ef | grep -i splunk and show us the output

[saranya_fmr]

@sloshburch

Splunk is running as stssplu1 , shown below,
stssplu1 35052 1 19 Dec08 ? 02:57:19 splunkd -p 8091 restart
stssplu1 35053 35052 0 Dec08 ? 00:00:00 [splunkd pid=35052] splunkd -p 8091 restart [process-runner]

Unable to add images so pasting the query result:

12/8/16
2:45:58.242 PM

12-08-2016 14:45:58.242 -0500 INFO ulimit - Limit: cpu time: unlimited
host = splnkdev01 source = /opt/common/apps/vendor/splunk/forwarder/var/log/splunk/splunkd.log sourcetype = ias_bass
12/8/16
2:45:58.242 PM

12-08-2016 14:45:58.242 -0500 INFO ulimit - Limit: user processes: 2060308 processes
host = splnkdev01 source = /opt/common/apps/vendor/splunk/forwarder/var/log/splunk/splunkd.log sourcetype = ias_bass
12/8/16
2:45:58.242 PM

12-08-2016 14:45:58.242 -0500 INFO ulimit - Limit: open files: 8192 files
host = splnkdev01 source = /opt/common/apps/vendor/splunk/forwarder/var/log/splunk/splunkd.log sourcetype = ias_bass
12/8/16
2:45:58.242 PM

12-08-2016 14:45:58.242 -0500 INFO ulimit - Limit: data file size: unlimited
host = splnkdev01 source = /opt/common/apps/vendor/splunk/forwarder/var/log/splunk/splunkd.log sourcetype = ias_bass
12/8/16
2:45:58.242 PM

12-08-2016 14:45:58.242 -0500 WARN ulimit - Core file generation disabled
host = splnkdev01 source = /opt/common/apps/vendor/splunk/forwarder/var/log/splunk/splunkd.log sourcetype = ias_bass
12/8/16
2:45:58.242 PM

12-08-2016 14:45:58.242 -0500 INFO ulimit - Limit: core file size: 0 bytes [hard maximum: unlimited]
host = splnkdev01 source = /opt/common/apps/vendor/splunk/forwarder/var/log/splunk/splunkd.log sourcetype = ias_bass
12/8/16
2:45:58.242 PM

12-08-2016 14:45:58.242 -0500 INFO ulimit - Limit: stack size: 10485760 bytes [hard maximum: unlimited]
host = splnkdev01 source = /opt/common/apps/vendor/splunk/forwarder/var/log/splunk/splunkd.log sourcetype = ias_bass
12/8/16
2:45:58.242 PM

12-08-2016 14:45:58.242 -0500 INFO ulimit - Limit: resident memory size: unlimited
host = splnkdev01 source = /opt/common/apps/vendor/splunk/forwarder/var/log/splunk/splunkd.log sourcetype = ias_bass
12/8/16
2:45:58.242 PM

12-08-2016 14:45:58.242 -0500 INFO ulimit - Limit: data segment size: unlimited
host = splnkdev01 source = /opt/common/apps/vendor/splunk/forwarder/var/log/splunk/splunkd.log sourcetype = ias_bass
12/8/16
2:45:58.242 PM

12-08-2016 14:45:58.242 -0500 INFO ulimit - Limit: virtual address space size: unlimited
host = splnkdev01 source = /opt/common/apps/vendor/splunk/forwarder/var/log/splunk/splunkd.log sourcetype = ias_bass

[sloshburch]

Why is the sourcetype sourcetype = is_bs and not splunkd? I might be confused about what you pasted.

[ddrillic]

I ran this query - index=_internal source=*splunkd.log ulimit open files.

Splunk shows in the UI 16384, but on the server I see -

-bash-4.1$ ulimit -n
1024

Strange!