Getting Data In

Troubleshooting why only part of a log file was imported

jwinderDDS
Path Finder

I have an application that creates a separate log file every time a major process runs. I discovered a single log file where the top several lines were not imported. A sample of the log files look like:

06/22/2015 20:30:06   > Header line 1
06/22/2015 20:30:06   > Header line 2
06/22/2015 20:30:06   > Header line 2
06/22/2015 20:30:06   > starting [SOMEPROC1]....
06/22/2015 20:30:06   > proc1 step 1
06/22/2015 20:30:07   > proc1 step 2
...
06/22/2015 20:30:07   > starting [SOMEPROC2]...
06/22/2015 20:30:07   > proc2 step 1
06/22/2015 20:30:30   > proc2 step 2
...

Splunk does not have any of the lines before the 'proc2 step2', starting at 20:30:30. Also, as far as I've been able to tell, this is the only time a log file hasn't been imported in its entirety. I did find the following in the UniversalForwarder's splunkd.log file:

06-22-2015 20:30:14.025 -0400 ERROR TcpOutputFd - Read error. An existing connection was forcibly closed by the remote host.
06-22-2015 20:30:14.025 -0400 INFO  TcpOutputProc - Connection to XXX.XXX.XXX.XXX:9997 closed. Read error. An existing connection was forcibly closed by the remote host.
06-22-2015 20:30:14.199 -0400 INFO  TcpOutputProc - Connected to idx=XXX.XXX.XXX.XXX:9997

This error occurs exactly between the lines that are missing and the lines Splunk has. My current understanding is the Universal Forwarder will queue events if it can't talk to the indexer and forward those events when the connection is re-established. Is this incorrect or do I have some other issue going on?

How do I go about troubleshooting why this happened?

Thank you in advance,

Jeremy

0 Karma
1 Solution

jnicholsenernoc
Path Finder

If you aren't using useACK=true in your outputs.conf, that may be why. They were in flight before the forwarder detected the receiver was down.

View solution in original post

jnicholsenernoc
Path Finder

If you aren't using useACK=true in your outputs.conf, that may be why. They were in flight before the forwarder detected the receiver was down.

jwinderDDS
Path Finder

This does seem to be the case. Thank you!

I have been able to verify that this has happened to roughly 40 files in the last 5 days. What is the best method of reindexing these files to pick up the missing events?

Thank you again,

Jeremy

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...