Thank you for the response. Unfortunately I have some problems.
I configured my cluster to archive data on s3 with shuttl. I installed Shuttl on all nodes. My data are really archived on s3 bucket but there is a problem with archiviation path.
Any path values I set as "archivePath" in archiver.xml( /myrootpath or myrootpath or or / ), Shuttl store my data in a root path "//" within my s3 bucket.
The result is like a directory without name, in the root of s3 bucket, that contains the directories structure created by shuttle.
For example if I set /myrootpath in archiver.xml, I find on s3 splunk buckets of type "s3://my_s3_bucket//myrootpath/archive_data/my_cluster/splunkIndex01/mytestdb/db_1397659802_1397655448_19_B95021DE-89AB-4A9D-B924-575736C54B81"
In the shuttl log I found these:
2014-04-16 18:56:22,037 INFO com.splunk.shuttl.archiver.archive.BucketFreezer: will="Attempting to archive bucket" index="mytestdb" path="/opt/splunk/var/lib/splunk/mytestdb/colddb/rb_1397667252_1397662059_21_B95021DE-89AB-4A9D-B924-575736C54B81"
2014-04-16 18:56:22,744 INFO com.splunk.shuttl.archiver.archive.BucketFreezer: will="Attempting to archive bucket" index="mytestdb" path="/opt/splunk/var/lib/splunk/mytestdb/db/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82"
2014-04-16 18:56:25,643 INFO com.splunk.shuttl.archiver.archive.recovery.FailedBucketsArchiver: will="Archiving failed buckets" failed buckets="[LocalBucket [getDirectory()=/opt/splunk/shuttl_archiver/data/safe-buckets/mytestdb/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82, getName()=db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82, getIndex()=mytestdb, getFormat()=SPLUNK_BUCKET, getPath()=/opt/splunk/shuttl_archiver/data/safe-buckets/mytestdb/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82, getEarliest()=Wed Apr 16 17:22:40 CEST 2014, getLatest()=Wed Apr 16 18:54:12 CEST 2014, getSize()=15912], LocalBucket [getDirectory()=/opt/splunk/shuttl_archiver/data/safe-buckets/mytestdb/rb_1397667252_1397662059_21_B95021DE-89AB-4A9D-B924-575736C54B81, getName()=rb_1397667252_1397662059_21_B95021DE-89AB-4A9D-B924-575736C54B81, getIndex()=mytestdb, getFormat()=SPLUNK_BUCKET, getPath()=/opt/splunk/shuttl_archiver/data/safe-buckets/mytestdb/rb_1397667252_1397662059_21_B95021DE-89AB-4A9D-B924-575736C54B81, getEarliest()=Wed Apr 16 17:27:39 CEST 2014, getLatest()=Wed Apr 16 18:54:12 CEST 2014, getSize()=16379]]"
2014-04-16 18:56:29,649 INFO com.splunk.shuttl.archiver.archive.BucketShuttlerRunner: will="Archiving bucket" bucket="LocalBucket [getDirectory()=/opt/splunk/shuttl_archiver/data/safe-buckets/mytestdb/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82, getName()=db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82, getIndex()=mytestdb, getFormat()=SPLUNK_BUCKET, getPath()=/opt/splunk/shuttl_archiver/data/safe-buckets/mytestdb/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82, getEarliest()=Wed Apr 16 17:22:40 CEST 2014, getLatest()=Wed Apr 16 18:54:12 CEST 2014, getSize()=15912]"
2014-04-16 18:56:30,700 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath%2Farchive_data%2Fmy_cluster%2FsplunkIndex02%2Fmytestdb%2Fdb_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82%2FCSV' - Unexpected response code 404, expected 200
2014-04-16 18:56:30,701 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath%2Farchive_data%2Fmy_cluster%2FsplunkIndex02%2Fmytestdb%2Fdb_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82%2FCSV' - Received error response with XML message
2014-04-16 18:56:31,089 INFO com.splunk.shuttl.archiver.archive.ArchiveBucketTransferer: will="attempting to transfer bucket to archive" bucket="LocalBucket [getDirectory()=/opt/splunk/shuttl_archiver/data/format-export-dir/mytestdb/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82/SPLUNK_BUCKET/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82, getName()=db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82, getIndex()=mytestdb, getFormat()=CSV, getPath()=/opt/splunk/shuttl_archiver/data/format-export-dir/mytestdb/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82/SPLUNK_BUCKET/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82, getEarliest()=Wed Apr 16 17:22:40 CEST 2014, getLatest()=Wed Apr 16 18:54:12 CEST 2014, getSize()=15912]" destination="/myrootpath/archive_data/my_cluster/splunkIndex02/mytestdb/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82/CSV"
2014-04-16 18:56:31,095 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Prepare transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/opt/splunk/shuttl_archiver/data/format-export-dir/mytestdb/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82/SPLUNK_BUCKET/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82, getName()=db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82, getIndex()=mytestdb, getFormat()=CSV, getPath()=/opt/splunk/shuttl_archiver/data/format-export-dir/mytestdb/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82/SPLUNK_BUCKET/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82, getEarliest()=Wed Apr 16 17:22:40 CEST 2014, getLatest()=Wed Apr 16 18:54:12 CEST 2014, getSize()=15912], remoteTemp=/myrootpath/temporary_data/splunkIndex02/myrootpath/archive_data/my_cluster/splunkIndex02/mytestdb/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82/CSV, dst=/myrootpath/archive_data/my_cluster/splunkIndex02/mytestdb/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82/CSV]"
2014-04-16 18:56:31,230 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath%2Farchive_data%2Fmy_cluster%2FsplunkIndex02%2Fmytestdb%2Fdb_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82%2FCSV' - Unexpected response code 404, expected 200
2014-04-16 18:56:31,230 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath%2Farchive_data%2Fmy_cluster%2FsplunkIndex02%2Fmytestdb%2Fdb_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82%2FCSV' - Received error response with XML message
2014-04-16 18:56:31,608 INFO com.splunk.shuttl.archiver.archive.BucketShuttlerRunner: will="Archiving bucket" bucket="LocalBucket [getDirectory()=/opt/splunk/shuttl_archiver/data/safe-buckets/mytestdb/rb_1397667252_1397662059_21_B95021DE-89AB-4A9D-B924-575736C54B81, getName()=db_1397667252_1397662059_21_B95021DE-89AB-4A9D-B924-575736C54B81, getIndex()=mytestdb, getFormat()=SPLUNK_BUCKET, getPath()=/opt/splunk/shuttl_archiver/data/safe-buckets/mytestdb/rb_1397667252_1397662059_21_B95021DE-89AB-4A9D-B924-575736C54B81, getEarliest()=Wed Apr 16 17:27:39 CEST 2014, getLatest()=Wed Apr 16 18:54:12 CEST 2014, getSize()=16379]"
2014-04-16 18:56:31,741 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath' - Unexpected response code 404, expected 200
2014-04-16 18:56:31,741 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath' - Received error response with XML message
2014-04-16 18:56:31,988 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath%2Farchive_data%2Fmy_cluster%2FsplunkIndex01%2Fmytestdb%2Fdb_1397667252_1397662059_21_B95021DE-89AB-4A9D-B924-575736C54B81%2FCSV' - Unexpected response code 404, expected 200
2014-04-16 18:56:31,989 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath%2Farchive_data%2Fmy_cluster%2FsplunkIndex01%2Fmytestdb%2Fdb_1397667252_1397662059_21_B95021DE-89AB-4A9D-B924-575736C54B81%2FCSV' - Received error response with XML message
2014-04-16 18:56:32,024 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath%2Ftemporary_data' - Unexpected response code 404, expected 200
2014-04-16 18:56:32,024 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath%2Ftemporary_data' - Received error response with XML message
In spite of this problem the Thaw and flush process of splunk buckets from search head interface works well.
I hope you can give me an answer as soon as possible.
Thanks in advance
... View more