The problem is that Splunk doesn't terminate all of its scripted inputs on all OS'es (i.e. ubunto or more specifically OS'es where /bin/sh points to dash or other shells where the command in shell -c "exec command" doesn't take over the process id of the shell which it was started with).
This is a known issue ( http://splunk-base.splunk.com/answers/28733/scripted-input-without-a-shell ) and should be resolved soon.
What's happening in your case:
Shuttl is trying to bind its own server to the host and port that you can configure in shuttl/conf/splunk.xml. Since the Shuttl server from an old Splunk process is still running, this cannot happen.
What you can do about it:
After each restart of Splunk, kill the Shuttl process. You'll be killing the Shuttl process that started for your last Splunk process.
Killing Shuttl is "safe". Shuttl is designed to fail and can always resume/recover from whatever state it was in.
How you do it [unix]:
- Get the Shuttl process id with either (
ps -ef | grep shuttl ) or ( lsof -i :9099 ) where 9099 is the default port for shuttl.
- Kill the process id (pid) with (
kill shuttl_pid).
I hope this helps and I'll update Shuttl as soon as Splunk has fixed the issue of not killing scripted inputs.
