Since upgrade from version 6.3.2 to 6.4, we are getting this problem. Search stuck at point of time and doesn't progress. From logs I found that the timestamp at which buckets rollover from hot to warm, search stuck at that timestamp and doesn't progress. This is not the case with all bucket rollovers but where ever there is below log message appears there is problem in search for buckets in that timespan.
INFO DatabaseDirectoryManager - idx=networks_syslog Writing a bucket manifest in hotWarmPath='/opt/splunk/datastore/networks_syslog/db', pendingBucketUpdates=0 . Reason='Buckets were rebuilt or tsidx-minified (bucket_count=1).'
And we DO NOT have tsidx reduction enabled. Also the '|dbinspect index=networks_syslog ' shows tsidx 'full'.
But I suspect there is some problem with indexing/tsidx files consistencies which is causing search stuck.
... View more