After simply ignoring this issue for quite a while, it has become relevant again, since we are currently in the process of upgrading from Splunk v5 to v6. One of my colleagues found a workaround which appears to partially solve our problems. I am posting this workaround for future reference and peer review.
1: Enable SRC subsystem
First, enable the Splunk SRC subsystem in AIX and instruct it to use the non-privileged useraccount (we'll use splunk , in this case):
/opt/splunk/uf/bin/splunk enable boot-start -user splunk
2: Remove SPLUNK_OS_USER limitation
Now, you need to remove the SPLUNK_OS_USER line in splunk-launch.conf . This will be added automatically by the previous command. If you don't, you won't be able to run the splunk start command as a user other than the one you specified earlier (and this will still fail due to the root cause of the inittab problem not being solved).
That's it. You will now be able to either use the default AIX startsrc -g splunk and stopsrc -g splunk commands and even run the binary as the root user. The daemon will drop privileges to the previously specified useraccount.
What you (still) won't be able to do
You still won't be able to run the splunk start and splunk stop commands directly as the non-privileged account. This may be a problem for some.
Testcases
After performing the actions in 1 and 2, the following testcases were carried out sequentially:
Starting point, Splunk is not running:
[root@bdaix71]:/root # oslevel -s
7100-03-01-1341
[root@bdaix71]:/root # /opt/splunk/uf/bin/splunk version
Splunk Universal Forwarder 6.2.1 (build 245427)
[root@bdaix71]:/root # ps -ef|grep [s]plunk
(no output)
[root@bdaix71]:/root # lssrc -g splunk
Subsystem Group PID Status
splunkd splunk inoperative
Start Splunk as root using the AIX SRC subsystem:
[root@bdaix71]:/root # startsrc -g splunk
0513-059 The splunkd Subsystem has been started. Subsystem PID is 4849824.
[root@bdaix71]:/root # ps -ef|grep [s]plunk
splunk 4849824 1900760 0 12:19:37 - 0:00 /opt/splunk/uf/bin/splunk _internal_exec_splunkd
splunk 11141354 4849824 120 12:19:42 - 0:00 /opt/splunk/uf/bin/splunkd btool server list general --no-log
[root@bdaix71]:/root # lssrc -g splunk
Subsystem Group PID Status
splunkd splunk 4849824 active
Stop Splunk as root using the AIX SRC subsystem:
[root@bdaix71]:/root # stopsrc -g splunk
0513-044 The splunkd Subsystem was requested to stop.
[root@bdaix71]:/root # lssrc -g splunk
Subsystem Group PID Status
splunkd splunk inoperative
[root@bdaix71]:/root # ps -ef|grep [s]plunk
(no output)
Start Splunk as root using direct call to binary:
[root@bdaix71]:/root # /opt/splunk/uf/bin/splunk start
Splunk> CSI: Logfiles.
Checking prerequisites...
Checking mgmt port [8089]: open
Checking conf files for problems...
Done
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
0513-059 The splunkd Subsystem has been started. Subsystem PID is 13762732.
Done
[root@bdaix71]:/root # ps -ef|grep [s]plunk
splunk 8781916 13762732 0 12:23:31 - 0:00 [splunkd pid=13762732] splunkd --nodaemon -p 8089 _internal_exec_splunkd process-runner]
splunk 13762732 1900760 2 12:21:55 - 0:21 splunkd --nodaemon -p 8089 _internal_exec_splunkd
[root@bdaix71]:/root # lssrc -g splunk
Subsystem Group PID Status
splunkd splunk 13762732 active
Stop Splunk as root using direct call to binary:
[root@bdaix71]:/root # /opt/splunk/uf/bin/splunk stop
Stopping splunkd...
Shutting down. Please wait, as this may take a few minutes.
0513-044 The splunkd Subsystem was requested to stop.
.
Stopping splunk helpers...
Done.
[root@bdaix71]:/root # ps -ef|grep [s]plunk
(no output)
[root@bdaix71]:/root # lssrc -g splunk
Subsystem Group PID Status
splunkd splunk inoperative
Login as the splunk non-privileged user and attempt to start the daemon (this still fails):
[root@bdaix71]:/root # su - splunk
[splunk@bdaix71]:/opt/splunk $ /opt/splunk/uf/bin/splunk start
Splunk> CSI: Logfiles.
Checking prerequisites...
Checking mgmt port [8089]: open
Checking conf files for problems...
Done
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
reading /etc/inittab: Permission denied
[splunk@bdaix71]:/opt/splunk $ ^D
[root@bdaix71]:/root # ps -ef|grep [s]plunk
(no output)
[root@bdaix71]:/root # lssrc -g splunk
Subsystem Group PID Status
splunkd splunk inoperative
Login as the splunk non-privileged user and attempt to stop the daemon (this also fails). Note that the daemon has been started between this testcase and the previous one:
[root@bdaix71]:/root # su - splunk
[splunk@bdaix71]:/opt/splunk $ /opt/splunk/uf/bin/splunk status
splunkd is running (PID: 13762732).
splunk helpers are running (PIDs: 8781916).
[splunk@bdaix71]:/opt/splunk $ /opt/splunk/uf/bin/splunk stop
reading /etc/inittab: Permission denied
[splunk@bdaix71]:/opt/splunk $ ps -ef|grep [s]plunk
splunk 8781916 13762732 0 12:23:31 - 0:00 [splunkd pid=13762732] splunkd --nodaemon -p 8089 _internal_exec_splunkd process-runner]
splunk 13762732 1900760 3 12:21:55 - 0:22 splunkd --nodaemon -p 8089 _internal_exec_splunkd
Reboot server and verify Splunk is running (after reconnecting):
[root@bdaix71]:/root # ps -ef|grep [s]plunk
splunk 7078106 3735728 0 12:52:01 - 0:11 splunkd --nodaemon -p 8089 _internal_exec_splunkd
splunk 9699386 7078106 0 12:55:00 - 0:00 [splunkd pid=7078106] splunkd --nodaemon -p 8089 _internal_exec_splunkd process-runner]
[root@bdaix71]:/root # lssrc -g splunk
Subsystem Group PID Status
splunkd splunk 7078106 active
... View more