I've been experimenting some out of memory issues in my server lately, basically the oom-killer
is called and one or more processes are killed. Among the processes that get killed there's always splunkforwarder.
After some testing, I decided to remove splunkforwarder from my server's boot and all problems stopped. If at any point in time I start the process, I get a new oom-killer issue.
Server is a small instance in amazon's ec2, using Ubuntu 12.04 LTS. This are my deploy commands:
/opt/splunkforwarder/bin/splunk start --accept-license
/opt/splunkforwarder/bin/splunk install app ... -auth admin:changeme
/opt/splunkforwarder/bin/splunk login -auth admin:changeme
/opt/splunkforwarder/bin/splunk edit user admin -password df5...f13
/opt/splunkforwarder/bin/splunk list forward-server
/opt/splunkforwarder/bin/splunk add monitor /var/log/apache2/error.log
The questions are:
This issue is generally when you have generalized a path to monitor.
Example [monitor:///var/logs/*]
This causes the issue. I had faced the same when we had added a generalized path (for oracle logs) the splunkd process was taking up heavy memory.
After changing path to specific log file to be monitored the memory usage settled down.
Example : [monitor:///var/log/messages]
Give it a go, hope it helps.
Down voted since I'm only monitoring one file:
/opt/splunkforwarder/bin/splunk add monitor /var/log/apache2/error.log
The good thing is that your comment lets me know that this is a splunk bug. It shouldn't take more memory to monitor a large number of files.
@Splunk developers: Please fix.
Not that I'm aware of.
Have you looked at the 5.0.2 version. It's been out for some time now.
Any ideas on what this could be about? Is this a known issue? When should I expect a fix? Anything I can do to help with testing the fix?
dpkg -i splunkforwarder-5.0.1-143156-linux-2.6-amd64.deb
Amazon ec2 AMI: ubuntu/images/ebs/ubuntu-precise-12.04-amd64-server-20121001
what version are you running?