The crash.log in the splunk forwarder has reflected Segmentation fault on starting up the process. How to resolve this configuration?

[terenceyhleung]

The splunk forwarder has been crash with segmentation fault when start the process in the AIX environment.

It also stated that
"Splunk may not work due to small resident memory size limit!"

The following is the return for the ulimit -a in the AIX environment.
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 131072
stack(kbytes) 32768
memory(kbytes) 32768
coredump(blocks) unlimited
nofiles(descriptors) 2000
threads(per process) unlimited
processes(per user) unlimited

Please advise which configuration should be change to fix the issue.

[dshakespeare_sp]

The AIX defaults are typically are not very generous on max file size (fsize) and resident memory size (rss).raise these limits for the user running splunk.

The Data Segment Size (ulimit -d) needs to be at least 1 GB (1073741824 bytes)

The Resident Memory Size (ulimit -m) needs to be at least :
512MB (536870912 bytes) for a Universal Forwarder
1 GB (1073741824 bytes) for a Indexer

Max No Of Open Files (ulimit -n) should be increased to at least 8192

File Size Limit (ulimit -f) should be set to unlimited (-1)

If these are not set high enough you will see errors in splunkd.log:
03-11-2015 09:34:42.631 +0100 INFO ulimit - Limit: virtual address space size: unlimited
03-11-2015 09:34:42.632 +0100 INFO ulimit - Limit: data segment size: 134217728 bytes [hard maximum: unlimited]
03-11-2015 09:34:42.632 +0100 WARN ulimit - Splunk may not work due to small data segment limit! <<<<<<<<<<<
03-11-2015 09:34:42.632 +0100 INFO ulimit - Limit: resident memory size: 33554432 bytes [hard maximum:
03-11-2015 09:34:42.632 +0100 WARN ulimit - Splunk may not work due to small resident memory size limit! <<<<<<<<<<<
03-11-2015 09:34:42.632 +0100 INFO ulimit - Limit: stack size: 33554432 bytes [hard maximum: 4294967296 bytes]
03-11-2015 09:34:42.632 +0100 INFO ulimit - Limit: core file size: 0 bytes
03-11-2015 09:34:42.632 +0100 WARN ulimit - Core file generation disabled
03-11-2015 09:34:42.632 +0100 INFO ulimit - Limit: data file size: unlimited
03-11-2015 09:34:42.632 +0100 INFO ulimit - Limit: open files: 4096 files [hard maximum: unlimited] <<<<<<<<<<<
03-11-2015 09:34:42.632 +0100 INFO ulimit - Limit: cpu time: unlimited
03-11-2015 09:34:42.632 +0100 INFO ulimit - Limit: data file size: 1073741312 bytes
03-11-2015 09:48:42.632 +0100 WARN ulimit - Splunk may not work due to low file size limit <<<<<<<<<<

These values are set in /etc/security/limits on AIX on a per user basis
These values need to defined as 512 byte blocks
eg
rss = 1048576 this is 536870912 bytes or 512 mB
data = 2097152 this is 1073741824 bytes or 1gB
fsize = -1 this is unlimited

This can also be set with smit (smitty chuser)
Be careful to use
Soft DATA segment
Soft RSS size

Again these values are in 512 byte blocks HOWEVER ulimit reports these in kB not 512b Blocks

A Worked Example
1. Login as root
2. # smitty chuser
Change the values for
Soft DATA segment [2097152]
Soft RSS size [1048576]
Soft NOFILE descriptors [8192]
Soft FILE size [-1]
Save and commit changes.
This basically just edits /etc/security/lmits:

Please be aware that need need to change the ulimit for the user that Splunkd runs under
Do not be tempted to change values to unlimited