Getting Data In

Why do forwarders go down with Unexpected EOF messages such as "FATAL ProcessRunner - Unexpected EOF from process runner child"?

ddrillic
Ultra Champion

Three of our forwarders went down today saying -

11-17-2016 15:39:33.525 -0600 INFO  HttpPubSubConnection - Running phone uri=/services/broker/phonehome/connection_10.123.200.143_8089_dbslt0080.uhc.com_dbslt0080_DC816E54-5F68-4259-BBE3-A97F24AD2EA8
11-17-2016 15:39:35.015 -0600 FATAL ProcessRunner - Unexpected EOF from process runner child!
11-17-2016 15:39:35.015 -0600 ERROR ProcessRunner - helper process seems to have died (child killed by signal 15: Terminated)!

What can the cause be?

Tags (2)
0 Karma

vincenteous
Communicator

Hi ddrillic ,

What OS are your Splunk running on top of? Is there any server management software running on the same server? I've had a similar experience like this when my Splunk was running on top of Red Hat with heavy load. If you are running Linux too, you may check from dmesg or /var/log/messages to see whether the similar type of error is recorded there (about terminating a process). If it exists, you may want to recheck the policies of your OS regarding process management.

I hope this could become your reference for troubleshooting.

kannu
Communicator

works for me ,
In my case some other person have killed the process (splunk) through ansible script .

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...