Getting Data In

Why Splunk Web UI shows "Check space and other issues that may cause indexer to block" and "Rawdata may be corrupted"?

Bizfinx_sysmon
New Member

Dear Support,

I have 2 messages on the Splunk web interface:
"skipped indexing of internal audit events will keep dropping events until indexer congestion is remedied. Check space and other issues that may caused indexer to block"
"Error in 'databasePartitionPolicy': Failure to read 1 event(s) from rawdata in bucket'_internal-1679-C10009BFA-1DE1-1A491-8895-E35E6F221168'. Rawdata maybe corrupted, see search.logs

I have tried to search under the following directory:
/opt/splunk/var/run/splunk/
where there are lots of directory with search.log.

I have tried to look at some of them and it seems to be ok.

Can someone from support advise on the above?

0 Karma

grijhwani
Motivator

Without looking at your specific setup, and/or having sight of specific error messages in the log, what you have told us above is suggestive of one of three things:

  1. Your index disk is full (although experience teaches me to expect a different message)
  2. Your index disk is experiencing errors which could be slowing throughput because of read/write retries (although again, if it is persistent I would expect a more explanatory error)
  3. Your indexer is quite simply over-taxed (through memory exhaustion and excessive page swapping, or CPU exhaustion) or has reached the bandwidth cap you have set (or that it has defaulted to)

I'd be inclined to look for a system problem to begin with.

0 Karma

Bizfinx_sysmon
New Member

Hi grijhwani,

Thanks for the update. FYI, I do not have such message anymore.

But I have a bigger issue now. I'm now not able to view my past data..
But I have check that the data seems to be there but Splunk is not able to link it back.

Can someone help?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...