My Splunk instance is constantly indexing data 24*7, but I've noticed some gaps in the indexed data timeline recently. I have also noticed that data I could search on yesterday is not being returned today. This doesn't happen consistently, but regularly enough to cause concern. I looked in splunkd.log and index=_internal to ensure that the buckets have not rotated out of the DB, and also confirmed that the buckets spanning the time period of the gap are present and in good shape. What else can I do to track down this missing data?
In splunkd.log I see the following:
05-07-2011 05:44:45.466 +0000 WARN MetaData - /opt/splunk/var/lib/splunk/apache/db/hot_v1_59/Hosts.data: attempting safeService to attempt to fix up metadata
My environment consists of 4 indexers running 4.2, 300 UF instances (also 4.2) and a standalone deployment server, also 4.2. We use the deployment server to manage the configs of all instances.
... View more