Monitoring Splunk

How to resolve error when Splunkd intermittently crashes while streaming telemetry data on Universal Forwarder: "ProcessRunner: No such file or directory"?

prajnaamey
Engager

Hello!

I’m working on streaming telemetry data to Splunk. I use Splunk Universal Forwarder v7 x86_64 to capture and stream data to Splunk Enterprise 8.

I use the script:// to capture data and run them at certain specified intervals. The data is being successfully streamed to the server. But, intermittently, splunkd (SUF) crashes, and I see the following error in my splunkd.log.

06-02-2020 17:12:27.975 -0700 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='/opt/splunkforwarder/var/log/splunk/btool.log'.
06-02-2020 17:12:27.993 -0700 INFO  WatchedFile - Will begin reading at offset=1182 for file='/opt/splunkforwarder/var/log/splunk/splunkd-utility.log'.
06-02-2020 17:12:56.832 -0700 INFO  ScheduledViewsReaper - Scheduled views reaper run complete. Reaped count=0 scheduled views
06-02-2020 17:30:37.696 -0700 WARN  TailReader - Could not send data to output queue (parsingQueue), retrying...
06-02-2020 17:53:37.315 -0700 ERROR ProcessRunner - Error from ProcessRunner helper process:  ERROR - Failed opening "": No such file or directory
06-02-2020 17:53:37.316 -0700 ERROR ProcessRunner - Error from ProcessRunner helper process:  terminate called after throwing an instance of 'EventLoopException'
06-02-2020 17:53:37.316 -0700 ERROR ProcessRunner - Error from ProcessRunner helper process:    what():  Main Thread: about to throw an EventLoopException: error from EventLoop poll: No such file or directory
06-02-2020 17:53:37.676 -0700 FATAL ProcessRunner - Unexpected EOF from process runner child!

I have tried to grok through Splunk answers and on Google; but, I couldn’t find much documentation/articles on what file ProcessRunner was trying to open? Could someone help me or point me to the right channel to understand how I can fix this issue.

Here’s my inputs.conf ’s script stanzas:

[script://$SPLUNK_HOME/bin/scripts/<script-one>.py]
source = source-one
sourcetype = source-one
[script://$SPLUNK_HOME/bin/scripts/<script-two>.path]
source = source-two
sourcetype = source-two
interval = 60
[script://$SPLUNK_HOME/bin/scripts/<script-three>.path]
source = source-three
sourcetype = source-three
interval = 1800
[script://$SPLUNK_HOME/bin/scripts/<script-four>.path]
source = source-four
sourcetype = source-four
interval = 1800

Thank you!

Labels (2)
0 Karma

efika
Communicator

Is it possible that you didn't do a proper error handling in the py scripts themselves ?
Are you trying to read some data in the python scripts and they will abort not in a graceful way while not being able to read the data ?

0 Karma

prajnaamey
Engager

Thank you for your reply, efika!

My initital thought was that there was an unhandled exception in the script. I removed the .py script and was seeing the same error.

I'm in the process of testing each stanza by itself to see if the culprit is one of our scripts.

I observed this issue occurring in SUF 7.x and SUF 8.x. I have had the same scripts running for SUF 6.x (32-bit) and did not encounter any such error. The SUF 6.x machines have been running for months now using the same scripts.

If my understanding is right - ExecProcessor runs the script stanzas, and it doesn't cause a Splunk crash if an error occurs while executing the script. I wonder what ProcessRunner is and what it's trying to do?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...