How come Splunk is not picking up the first few lines (3-5 line) of our log files?

[aknsun]

Hi,

I have an issue where Splunk is not picking up the first few lines (3-5 line) of log files when doing a search. There is no customization done via the props and transforms.

I have also checked and didn't find any messages in $SPLUNK_HOME/var/log/splunk/splunkd.log on the forwarder that pointed to any issue of these lines being skipped.

Any suggestions?

Regards,

AKN.

[chrisyounger]

Hi @aknsun

I can't see anything obviously wrong with your log that would cause events to go missing.

The following things could be happening:

• The automatic datetime detection is not working properly for your timestamps and Splunk thinks the events are either in the future or very far in the past. Try running this search to identify if this is the cause: index ="whatever" source="path of the log file" earliest=0 latest=+10d
• You might be using a source or sourcetype that is discarding your events. Splunk out-of-the-box does come with some special configurations for some sourcetypes. You should run btool on the server to try and identify if this is the case. Example /opt/splunk/bin/splunk btool props list <sourcetype> --debug

Hope this helps.

[aknsun]

Hi @chrisyoungerjds

1. I checked the first option and the result seems to be the same. Some events are missing.
2. the sourcetype is log4j. So I believe that should be ok.

Regards,
AKN

[chrisyounger]

Hi aknsun, Are you able to share an example of the log file lines that are not displaying along with the search you are running?

[aknsun]

Search

Index = "index name" source = "path of the log file"

Search only returns the 3rd line in this case. The first 2 lines are not returned.

Log details (Masked here)
2019-01-23 04:18:04,537 INFO [pool-1-thread-1] Create ******** success.
2019-01-23 11:03:01,994 INFO [pool-1-thread-2] Create ******** success.
2019-01-23 11:37:14,436 INFO [pool-1-thread-3] Create ******** success.