Getting Data In

Splunk forwarder on windows server 2008R2 won't start

mhorn
New Member

I have a forwarder that was running fine for a couple days but I had to turn it off due to a system resources issue. I have since added more memory and when I try to start the service from the MS services GUI i get an error 1067. The event viewer is only showing me an event id of 7034, splunkforwarder service terminated unexpectedly...that's it. I have the service set to start as a local system account. Doing a Google search on those results return some MySQL results, and windows SCM service, which doesn't help me resolve the issue.

If I try to start it from the command line it says the port is available, but times out when it tries to start the splunkd.

Anybody have a clue what's causing this? I'm guessing I need to un-install/reinstall the forwarder. I just want to understand why all of a sudden it won't start anymore.

0 Karma

adnanwali
Engager

we also have this issue on windows 2008 R2 and windows 2012 R2 servers randomly splunk service dont start after server reboot. Windows event log report that The SplunkForwarder Service service terminated unexpectedly

Clean lock fixes the issue or server reboot also fixes but that not the solution if you have thousands of windows servers have splunk agents installed and they crash randomly

slgizmo
New Member

I would like to know why this happens. Granted this command did release the lock on the file and I was successful at starting the service but why is this randomly happening so that I no longer need to use the band-aid.

0 Karma

dstambaugh
Explorer

There appears to be a lock on a config file. Either restart your Windows server or run this from the command line:

C:\Program Files\SplunkUniversalForwarder\bin\splunk.exe clean locks

See also this link.

AaronMoorcroft
Communicator

Thank you, this worked a treat for me when rebooting the server was not an option 🙂

0 Karma

bmacias84
Champion

Question if you reboot the server does it start working again? I know their have been bugs with Ephemeral port exhaustion depending on your version.

0 Karma

mhorn
New Member

sorry, just did. completely forgot that log. In the log it's just showing the following.

0700 FATAL loader - Timed out waiting for config lock; see splunkd_stderr.log for details. Exiting.

I could not find splunkd_stderr.log on the server or the primary indexer.

0 Karma

mgaraventa_splu
Splunk Employee
Splunk Employee

On Windows systems it's normal not to find splunkd_stderr.log, as Windows Services don't have a stderr at all. So this is the most likely reason why you cannot find that log file.

0 Karma

gregbujak
Path Finder

Hi, let me start by asking, have you checked the splund.log?
$SPLUNK_HOME/var/log/splunk/splunkd.log

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...