Hi,
I was working on a puppet module to update existing splunkforwarder from 6.1.* to 7.1.7. Installation went smooth but now the forwarder is failing to start with below error message.
systemd[1]: Failed to read PID from file /**/splunk/splunkd.pid: Invalid argument
Centos is the target machine.
splunk.service file
[Unit]
Description=Splunk Enterprise 7.1.7
After=network.target
Wants=network.target
[Service]
Type=forking
RemainAfterExit=False
User=root
Group=root
LimitNOFILE=65536
ExecStart=/**/splunk start --accept-license --answer-yes --no-prompt
ExecStop=/**/splunkstop
PIDFile=/**/splunkforwarder/var/run/splunk/splunkd.pid
Restart=always
[Install]
WantedBy=multi-user.target
# If you want to use $(systemctl [start|stop|restart] splunk) instead of splunkd ...
Alias=gsosplunk.service
And the puppet script involves below procedure.
Removing files in existing custom installation path of splunkforwarder.
Extracting the 7.1.7 tar ball package to the custom path.
Copying required certs for server connectivity.
Accepting licence and updating the permissions to be accessible by splunk "chown -R splunk:splunk" (This might not be required at all. Since the splunk will be started as root user. But I just thought of having it ).
Moving deploymentclient and web.conf files to the custom path.
For starting the splunk agent.
I just ensured that splunk.service is having necessary permissions
Reload the systemctl daemon and ensure that it is running via puppet code.
I came across couple of posts where the files are not having enough permission. But in my case, since the splunk is started with root privileges I dont see anything like that.
And on few occasions, I was able to successfully start the process (by removing PID file when the process is running) but i dont think its the proper way to start it.
... View more