Hi,
I installed the universal forwarder 6.1 for AIX.
splunkforwarder-6.1.1-207789-AIX-powerpc.tar
splunk@xxxxxx /opt/splunk > uname -a
AIX xxxxxx 1 6 00F783D04C00
The first time the forwarder was restarted I got this error:
splunk@xxxxxxxx /opt/splunk/splunkforwarder/bin > ./splunk start
Splunk> All batbelt. No tights.
Checking prerequisites...
Checking mgmt port [8089]: open
Checking conf files for problems...
Done
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
execve: Permission denied
I can't find anything about this in the splunk log files.
Best regards,
Anne Beate Arneberg
We had the same issue with all versions of 6.1.x , 6.2 and the last tried was 6.2.1.
We resorted to replacing all of our startup scripts with Wrapper scripts like the one recommended by pit4getit .
That solves the issue for us but not a recommended final fix due to the extra work of replacing the startup scripts.
If you run into issues with Splunk starting on 6.1.x and 6.2.x versions please try following steps:
lssrc -a | grep splunk
. if this returns any output, it means that we still have Splunk information in the object class. Typically this will be "splunkd". If this exists, it needs to be removed with rmssys -s splunkd
. lssrc -a | grep splunk
to ensure that "splunkd" is removed from the object class. chmod -R splunk /opt/splunkforwarder
. This will change the permissions of all the files under /opt/splunkforwarder to "splunk". find /opt/splunkforwarder \! -user splunk | xargs ls -al
. This will return NO results if all the files are owned by splunk. /opt/splunkforwarder/bin/splunk start
. Splunk should now start. sudo -u splunk
.There is actually a bug open for this issue: one for the 6.1.x releases (SPL-85092) and a clone for the 6.2.x releases (SPL-92963). The expected maintenance releases containing the fix are 6.1.7 and 6.2.2 respectively. Once they are released, please upgrade and make sure that you don't see issues anymore. If you should still face issues, then please file a new support case by providing:
A. a new diag
B. the history of actions perfomed with the according outputs.
and Splunk> Support will then further investigate the issue.
With these SPL's/bugs I had expected not to see this:
"execve: Permission denied
while running command /usr/bin/startsrc
Splunk boot-start is enabled. please use /usr/bin/startsrc -s splunkd to start splunk "
when I yesterday tried to migrate 6.0.5 UF's to 6.3.0. But the issue is still there. Is this something that was caused by the old installation, or is it still not fixed? It also occured in 6.1.1 and was said to be solved in 6.1.4 after fixing SPL-85092
Also, I installed 6.0.6 (last release for 6.0) splunkforwarder for AIX and it works fine.
Then I removed it and tried 6.1.0(first release for 6.1) version and it breaks there i.e the same execv:permission denied.
So it seems something happened while going from 6.0 to 6.1.
@HansWurscht
I am also facing same issue i.e execve: Permission denied when trying to run splunkforwarder on AIx.
I have tried splunkforwarder 6.1.2,6.1.3 and 6.1.4. But none of the versions have a fix for the issue. All of them still gives same error.
Hi,
did have same problem.
My problem was i did install first time the splunkforwarder with"splunk enable boot-start".
This makes entries in SRCsubsys of aix:
Check if entry exists:
odmget SRCsubsys | grep splunk
Get rid of entries with
sudo .../.splunk disable boot-start
Check if entry exists, and start splunk with wrapper script as user splunk, and never as user root like:
$ cat rc.splunk
#!/bin/ksh
### Script Variables
APPDIR=/usr/local/splunk/splunkforwarder/bin
APPNAME=splunk
RUN_USER=splunk
#export JAVA_HOME=/usr/java7_64
case "$1" in
start )
echo "Starting the ${APPNAME} Application"
su $RUN_USER -c ${APPDIR}/${APPNAME} start
;;
stop )
echo "stopping the Application"
su $RUN_USER -c ${APPDIR}/${APPNAME} stop
;;
restart )
echo "restarting the ${APPNAME} Application"
su $RUN_USER -c ${APPDIR}/${APPNAME} stop
su $RUN_USER -c ${APPDIR}/${APPNAME} start
;;
* )
echo "Usage: $0 COMMAND"
echo "---------------------------"
echo "start - Start ${APPNAME} instance in ${APPDIR}"
echo "stop - Stop ${APPNAME} instance in ${APPDIR}"
echo "restart - Stop ${APPNAME} instance in ${APPDIR}"
echo
exit 1
esac
Hope thie helps
cheers, pit
What is the issue? I'm doing a pilot on my AIX server farm, and out of the 20 or so, one failed. It actually ran before. And now it's not running. This is very strange.
This will be fixed in 6.1.4
Hi,
It did not help to change the file permissions to splunk. I tried to reinstall the 6.1.1 version and encountered the same problem during restart of splunk. I ended up with installing the 6.0.4 version which worked 🙂
We are running splunkforwarder-6.1.1-207789-AIX-powerpc.
We succesfully had it start and run for 2 day but today we get the exact same error as above.
We checked that all files and directories under .apps/splunk (where all our instances go) are owned by the correct splunk service account.
We are stumped. Any suggestions?
Hi annebeate,
check the permission of /opt/splunk
, all the files must be owned by the user splunk
in your case. If you started the forwarder before with the root
account, you will have some files owned by root
. Check the docs about Run Splunk as a different or non-root user
hope this helps ...
cheers, MuS