I have multiple crashes on my VM Linux servers "SUSE 12" that are running Splunk service in a cluster, mainly what is crashing are indexers and Search heads. We had different causes from the crash logs under Splunk which is Segmentation Fault and also on the var/log messages we see logs for crashes with a Segmentation fault. What can be monitored on the Server OS level to identify the root cause of the issue like what resources should be monitored to triage these crashes?
That means Splunk is consuming more memory than it is allowed by the OS, or it is exhausting all of the system memory and/or SWAP.
Ensure that ulimits are set correctly for your system:
https://docs.splunk.com/Documentation/Splunk/8.0.2/Installation/SystemRequirements#Considerations_re...
Increase the memory on your VM's or increase the size of your cluster, or decrease the workload.
If you found the suggestions to help, please "accept" the answer so that the community can benefit.