Splunk Dev

Splunk skipping some messages to read from file

ankithreddy777
Contributor

I have a log files updated in realtime. From past two years these files are ingested to splunk without issues. Suddenly I found a weird issue, where splunk skipping some messages in a file to ingest here and there . I found around 10 percent of the messages are skipped.

I am not sure where is the root cause. I can understant if it skips complete file, but its skipping messages here and there in a single file. Its happening for all files ingested from that source. No configs are changed.

I cannot search for any field value in the missing message in splunk.

Should I begin troubleshooting for problems on indexer side or forwarder side.

May I know what might cause such type of issue.

0 Karma

ankithreddy777
Contributor

Hi kamlesh ,
Thank you for your reply.
I checked disk space and errors in splunkd.
There are no errors.
I have observed that while searching for data, I can only get data from 17 indexers instead of 20 indexers. Search for current index does not show any results from remaining three indexers exactly from the date we observed data is missing.
But these three indexers are up and healthy and show results for other indexes.

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

Hi Ankithreddy777,

There might be any possibilities for this issue. But I think it should be below:

  • if you have recently started forwarding new events in the different index then check the existence of the index and check splunkd.log of the indexer.
  • It might be disk space or disk related issue.

you can troubleshoot the problem by following below link.

https://wiki.splunk.com/Community:TroubleshootingIndexing

Thanks

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...