Deployment Architecture

How to restore frozen data when it looks my buckets have corrupted files?

guimilare
Communicator

Hi all.

I'm running Splunk 6.3.2 in Linux machines.

I have to restore some old data to Splunk.
I've followed the steps described at http://docs.splunk.com/Documentation/Splunk/6.3.2/Indexer/Restorearchiveddata but I'm facing an error that I can't solve.

I've copied the frozen bucket db_1438454243_1438385486_4557 to /var/splunk/defaultdb/thaweddb/
After that, I executed the following command:

/opt/splunk/bin/splunk rebuild /var/splunk/defaultdb/thaweddb/db_1438454243_1438385486_4557 
USAGE: splunk rebuild <bucketPath> [<indexName>] [--ignore-read-error] [--no-log] The <indexName> parameter is ignored if provided. Please see 'splunk fsck' for more options.  This command is just a wrapper for 'splunk fsck'.

Redirecting to 'splunkd fsck' with args:
        repair --one-bucket --include-hots --bucket-path=/var/splunk/defaultdb/thaweddb/db_1438454243_1438385486_4557
--log-to--splunkd-log ERROR JournalSlice - Error reading compressed journal while streaming: bad gzip header, provider=/var/splunk/defaultdb/thaweddb/db_1438454243_1438385486_4557/rawdata/journal.gz ERROR BucketBuilder - Error reading rawdata: Error reading compressed journal while streaming: bad gzip header, provider=/var/splunk/defaultdb/thaweddb/db_1438454243_1438385486_4557/rawdata/journal.gz WARN  Fsck - Repair (entire bucket) idx= bucket='/var/splunk/defaultdb/thaweddb/db_1438454243_1438385486_4557' failed: (entire bucket) Rebuild for bkt='/var/splunk/defaultdb/thaweddb/db_1438454243_1438385486_4557' failed: Error reading rawdata: Error reading compressed journal while streaming: bad gzip header, provider=/var/splunk/defaultdb/thaweddb/db_1438454243_1438385486_4557/rawdata/journal.gz Rebuilding bucket failed

Inside /var/splunk/defaultdb/thaweddb/db_1438454243_1438385486_4557/rawdata there are only 3 files: slicesv2.dat, slicemin.dat and journal.gz.

It looks like the buckets have corrupted files, since the command file in linux five me that the journal.gz file is data.
Is that correct or I'm missing something?

Thank in advance.
Best regards.

0 Karma

Masa
Splunk Employee
Splunk Employee

Yes, sounds like your bucket is corrupted.

How about "gunzip journal.gz" Does this work without error? Most likely this will return error, too.
Unfortunately, if that's the case, there is no tool available recover the bucket.

somesoni2
Revered Legend
0 Karma
Get Updates on the Splunk Community!

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer at Splunk .conf24 ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...

Share Your Ideas & Meet the Lantern team at .Conf! Plus All of This Month’s New ...

Splunk Lantern is Splunk’s customer success center that provides advice from Splunk experts on valuable data ...

Combine Multiline Logs into a Single Event with SOCK: a Step-by-Step Guide for ...

Combine multiline logs into a single event with SOCK - a step-by-step guide for newbies Olga Malita The ...