I have a few sh scripts scheduled to run every few min and those stop recently, and print this error in the log. To work around is simple as bouncing the forwarder instance, but it only works for a day or two and die again. Anyone has any thoughts?
Couldn't start command "/app/splunkforwarder/etc/apps/Splunk_TA_nix/bin/xxx.sh": Resource temporarily unavailable
I am not a linux admin by any means, but I would look at your ulimits for the Splunk user that's running the forwarder. Maybe the scripts are finishing fast enough and your number of processes is growing until you hit those limits? Just a thought.