Splunk forwarder on windows server 2008R2 won't start

[mhorn]

I have a forwarder that was running fine for a couple days but I had to turn it off due to a system resources issue. I have since added more memory and when I try to start the service from the MS services GUI i get an error 1067. The event viewer is only showing me an event id of 7034, splunkforwarder service terminated unexpectedly...that's it. I have the service set to start as a local system account. Doing a Google search on those results return some MySQL results, and windows SCM service, which doesn't help me resolve the issue.

If I try to start it from the command line it says the port is available, but times out when it tries to start the splunkd.

Anybody have a clue what's causing this? I'm guessing I need to un-install/reinstall the forwarder. I just want to understand why all of a sudden it won't start anymore.

[adnanwali]

we also have this issue on windows 2008 R2 and windows 2012 R2 servers randomly splunk service dont start after server reboot. Windows event log report that The SplunkForwarder Service service terminated unexpectedly

Clean lock fixes the issue or server reboot also fixes but that not the solution if you have thousands of windows servers have splunk agents installed and they crash randomly

[slgizmo]

I would like to know why this happens. Granted this command did release the lock on the file and I was successful at starting the service but why is this randomly happening so that I no longer need to use the band-aid.

[dstambaugh]

There appears to be a lock on a config file. Either restart your Windows server or run this from the command line:

C:\Program Files\SplunkUniversalForwarder\bin\splunk.exe clean locks

See also this link.

[AaronMoorcroft]

Thank you, this worked a treat for me when rebooting the server was not an option 🙂

[bmacias84]

Question if you reboot the server does it start working again? I know their have been bugs with Ephemeral port exhaustion depending on your version.

[mhorn]

sorry, just did. completely forgot that log. In the log it's just showing the following.

0700 FATAL loader - Timed out waiting for config lock; see splunkd_stderr.log for details. Exiting.

I could not find splunkd_stderr.log on the server or the primary indexer.

[mgaraventa_splu]

On Windows systems it's normal not to find splunkd_stderr.log, as Windows Services don't have a stderr at all. So this is the most likely reason why you cannot find that log file.

[gregbujak]

Hi, let me start by asking, have you checked the splund.log?
$SPLUNK_HOME/var/log/splunk/splunkd.log