Hi, I am getting this error when I open one of my dashboards today.
" Error in 'databasePartitionPolicy': Failed to read 1 event(s) from rawdata in bucket 'main-xxxxxx'. Rawdata may be corrupt, see search.log."
this is what i see in search.log
02-26-2013 11:22:21.540 INFO DispatchCommand - Round Robin Threaded ProviderQueue: done reading from peer 'BP1LCSAP031'
02-26-2013 11:22:23.506 ERROR JournalSlice - Cannot seek to 74529344
02-26-2013 11:22:23.506 ERROR databasePartitionPolicy - Failed to read event at address=2329042 in rawdata directory: \reuxeuss019-f07\splunk_index\defaultdb\db\db_1361833650_1361568580_55\rawdata
02-26-2013 11:22:23.506 ERROR databasePartitionPolicy - Failed to read 1 event(s) from rawdata in bucket 'main~55~004CC9C7-AEAA-4C5A-B3C7-2B22F4A91F7D'. Rawdata may be corrupt, see search.log
02-26-2013 11:22:23.521 INFO IndexScopedSearch - PREAD_HISTOGRAM: usec_1_8=3718 usec_8_64=0 usec_64_512=0 usec_512_4096=0 usec_4096_32768=9
Any suggestions please?
You may need to manually run FSCK against your buckets, have a look here for the detail;
http://docs.splunk.com/Documentation/Splunk/latest/Indexer/HowSplunkstoresindexes#Troubleshoot_your_...
Also, if you store your buckets on another filesystem/partition make sure that there are no issues with permissions or the user that Splunk is running as can access them still.
You may need to manually run FSCK against your buckets, have a look here for the detail;
http://docs.splunk.com/Documentation/Splunk/latest/Indexer/HowSplunkstoresindexes#Troubleshoot_your_...
Also, if you store your buckets on another filesystem/partition make sure that there are no issues with permissions or the user that Splunk is running as can access them still.
running FSCK helped
No, I haven't tried a reboot yet but this was working fine till yesterday. Also, I see these as well in the indexing errors:
INFO databasePartitionPolicy - idx=_audit Moving from='hot_v1_48' to warm='write error on hot bucket'
» 2/26/13
11:46:04.961 AM
02-26-2013 11:46:04.961 +0000 ERROR databasePartitionPolicy - Unable to write raw: for idx=_audit, path='\reuxeuss019-f07\splunk_index\audit\db\hot_v1_48'
» 2/26/13
11:45:26.989 AM
02-26-2013 11:45:26.989 +0000 INFO databasePartitionPolicy - idx=_internal Moving from='hot_v1_67' to warm='write error on hot bucket'
tried a reboot of splunkd? this may rebuild corrupt sections.