Monitoring Splunk

Why are certain log events not getting indexed in Splunk 5.0.4 and how to troubleshoot?

strive
Influencer

Hi,

One of our customers is using Splunk 5.0.4. The log files are forwarded to indexer using Splunk Universal Forwarder.

The log in flow is like this:

Splunk UF on Devices --> Splunk UF in the product --> Indexer

The issue is: At times, some log events are not getting indexed and this leads to data inaccuracy in our metrics. Recently when they reported this issue, i took log files from them and indexed them in my local test bed. I was able to replicate the issue. Out of 5000 log events, 7 events did not enter the index. Similarly in other log file, out of 5085 log events, 13 events did not enter the index.

I checked following:

1. If log event length is on the higher side -- answer is No.

2. If some unreasonable junk characters are present in the log event -- answer is No.

3. If the log events are duplicate of other log events -- answer is No.

Could you suggest some pointers for me to troubleshoot this issue. Why some specific log lines are not getting indexed?

Note: This is not happening all the time. In last two weeks this has happened twice for around 10 log files.

Thanks

Strive

0 Karma
1 Solution

strive
Influencer

The log files had secondary header line starting with words s-ip|#Fields.

If the log lines had any field value(s) with s-ip as substring then those log lines were stripped off.
We had to modify our transforms.conf configurations to address this issue.

View solution in original post

strive
Influencer

The log files had secondary header line starting with words s-ip|#Fields.

If the log lines had any field value(s) with s-ip as substring then those log lines were stripped off.
We had to modify our transforms.conf configurations to address this issue.

strive
Influencer

The log files are not rolling.
We have set nullQueue for headers. This wont interfere with these log lines.

0 Karma

MuS
SplunkTrust
SplunkTrust

try to index the events again while running this script http://blogs.splunk.com/2011/01/02/did-i-miss-christmas-2/

0 Karma

pradeepkumarg
Influencer

Are the log files rolling? If so, check if the events are being missed for some reason while the log is being rolled.

0 Karma

MuS
SplunkTrust
SplunkTrust

any nullQueue in any transforms.conf which could interfere here? check with btool

strive
Influencer

Link to files
https://www.dropbox.com/s/5g8q4d40j5mwf2b/my_data.13.13.13.13_20140823_114500_1501?dl=0

[my_source_type]
SHOULD_LINEMERGE = false
TRANSFORMS-include = some transforms
TIME_PREFIX=^([^\t]*\t){2}
MAX_TIMESTAMP_LOOKAHEAD=35

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Would it be possible for you to share those events which are not getting indexed? (may after masking sensitive information), Also, the sourcetype definition (props.conf)?

0 Karma

strive
Influencer

Created a log file using the missing events alone and tried indexing this file. The events are not getting indexed, there are no errors in splunkd.log (enabled debug mode and checked). Manually verified every field in the log file, it all looks fine.

0 Karma

strive
Influencer

They are from same sourcetype. There is no commanlity.

0 Karma

jbouch03
Path Finder

Are they the same sourcetype or different? Also, is there any commonality among the events that are not getting indexed?

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...