Splunk skipping some messages to read from file

[ankithreddy777]

I have a log files updated in realtime. From past two years these files are ingested to splunk without issues. Suddenly I found a weird issue, where splunk skipping some messages in a file to ingest here and there . I found around 10 percent of the messages are skipped.

I am not sure where is the root cause. I can understant if it skips complete file, but its skipping messages here and there in a single file. Its happening for all files ingested from that source. No configs are changed.

I cannot search for any field value in the missing message in splunk.

Should I begin troubleshooting for problems on indexer side or forwarder side.

May I know what might cause such type of issue.

[ankithreddy777]

Hi kamlesh ,
Thank you for your reply.
I checked disk space and errors in splunkd.
There are no errors.
I have observed that while searching for data, I can only get data from 17 indexers instead of 20 indexers. Search for current index does not show any results from remaining three indexers exactly from the date we observed data is missing.
But these three indexers are up and healthy and show results for other indexes.

[kamlesh_vaghela]

Hi Ankithreddy777,

There might be any possibilities for this issue. But I think it should be below:

• if you have recently started forwarding new events in the different index then check the existence of the index and check splunkd.log of the indexer.
• It might be disk space or disk related issue.

you can troubleshoot the problem by following below link.

https://wiki.splunk.com/Community:TroubleshootingIndexing

Thanks