- Splunk Answers
- :
- Splunk Administration
- :
- Monitoring Splunk
- :
- HIgh Physcial Memory Usage in Splunk searchhead VM
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
HIgh Physcial Memory Usage in Splunk searchhead VM
Hi , I am currently facing this High Physical VM usage alert in splunk adhoc searchhead which is in VM environment (just did the HealthCheck). The specs are
Memory Used (MB) 17592186037778.105
Physical Memory Installed (MB) 15870.715
Memory used (%) 110846839841.7%
The THP aren't disabled and open files are 4096 against the recommended 8192. What could be the reason for this excessive memory usage and how to control it. Help would be highly appreciated.
Regards,
KJ
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@kartreddy4,
You can try referring this doc:
http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Troubleshootmemoryusage
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Well not having the THP and ulimits properly set could be the main issue
Check If this system is running Splunk as a user called "splunk," add the following to the bottom of the /etc/security/limits.conf file:
Should look like this your cat /opt/splunk/var/log/splunk/splunkd.log | grep ulimit
03-01-2017 22:08:01.857 +0000 INFO ulimit - Limit: virtual address space size: unlimited
03-01-2017 22:08:01.857 +0000 INFO ulimit - Limit: data segment size: unlimited
03-01-2017 22:08:01.857 +0000 INFO ulimit - Limit: resident memory size: unlimited
03-01-2017 22:08:01.857 +0000 INFO ulimit - Limit: stack size: 8388608 bytes [hard maximum: unlimited]
03-01-2017 22:08:01.857 +0000 INFO ulimit - Limit: core file size: 0 bytes
03-01-2017 22:08:01.857 +0000 WARN ulimit - Core file generation disabled
03-01-2017 22:08:01.857 +0000 INFO ulimit - Limit: data file size: unlimited
03-01-2017 22:08:01.857 +0000 INFO ulimit - Limit: open files: 65535 files
03-01-2017 22:08:01.857 +0000 INFO ulimit - Limit: user processes: 20480 processes
03-01-2017 22:08:01.857 +0000 INFO ulimit - Limit: cpu time: unlimited
03-01-2017 22:08:01.857 +0000 INFO ulimit - Linux transparent hugepage support, enabled="never" defrag="never"
splunk soft fsize unlimited
splunk hard fsize unlimited
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Please let me know if the answer was useful for you. If it was, accept it and upvote. If not, give us more input so we can help you with that
Adoption of RUM and APM at Splunk
Routing logs with Splunk OTel Collector for Kubernetes
Welcome to the Splunk Community!
