Hello!
I’m working on streaming telemetry data to Splunk. I use Splunk Universal Forwarder v7 x86_64 to capture and stream data to Splunk Enterprise 8.
I use the script://
to capture data and run them at certain specified intervals. The data is being successfully streamed to the server. But, intermittently, splunkd
(SUF) crashes, and I see the following error in my splunkd.log.
06-02-2020 17:12:27.975 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunkforwarder/var/log/splunk/btool.log'.
06-02-2020 17:12:27.993 -0700 INFO WatchedFile - Will begin reading at offset=1182 for file='/opt/splunkforwarder/var/log/splunk/splunkd-utility.log'.
06-02-2020 17:12:56.832 -0700 INFO ScheduledViewsReaper - Scheduled views reaper run complete. Reaped count=0 scheduled views
06-02-2020 17:30:37.696 -0700 WARN TailReader - Could not send data to output queue (parsingQueue), retrying...
06-02-2020 17:53:37.315 -0700 ERROR ProcessRunner - Error from ProcessRunner helper process: ERROR - Failed opening "": No such file or directory
06-02-2020 17:53:37.316 -0700 ERROR ProcessRunner - Error from ProcessRunner helper process: terminate called after throwing an instance of 'EventLoopException'
06-02-2020 17:53:37.316 -0700 ERROR ProcessRunner - Error from ProcessRunner helper process: what(): Main Thread: about to throw an EventLoopException: error from EventLoop poll: No such file or directory
06-02-2020 17:53:37.676 -0700 FATAL ProcessRunner - Unexpected EOF from process runner child!
I have tried to grok through Splunk answers and on Google; but, I couldn’t find much documentation/articles on what file ProcessRunner
was trying to open? Could someone help me or point me to the right channel to understand how I can fix this issue.
Here’s my inputs.conf
’s script stanzas:
[script://$SPLUNK_HOME/bin/scripts/<script-one>.py]
source = source-one
sourcetype = source-one
[script://$SPLUNK_HOME/bin/scripts/<script-two>.path]
source = source-two
sourcetype = source-two
interval = 60
[script://$SPLUNK_HOME/bin/scripts/<script-three>.path]
source = source-three
sourcetype = source-three
interval = 1800
[script://$SPLUNK_HOME/bin/scripts/<script-four>.path]
source = source-four
sourcetype = source-four
interval = 1800
Thank you!
Is it possible that you didn't do a proper error handling in the py scripts themselves ?
Are you trying to read some data in the python scripts and they will abort not in a graceful way while not being able to read the data ?
Thank you for your reply, efika!
My initital thought was that there was an unhandled exception in the script. I removed the .py
script and was seeing the same error.
I'm in the process of testing each stanza by itself to see if the culprit is one of our scripts.
I observed this issue occurring in SUF 7.x and SUF 8.x. I have had the same scripts running for SUF 6.x (32-bit) and did not encounter any such error. The SUF 6.x machines have been running for months now using the same scripts.
If my understanding is right - ExecProcessor runs the script stanzas, and it doesn't cause a Splunk crash if an error occurs while executing the script. I wonder what ProcessRunner is and what it's trying to do?