I have a cluster with Splunk 5.2, consisting of 1 master node, 5 indexer node and 1 search head and I'm trying to get shuttl to archive frozen bucket, avoiding, in this way, buckets duplication problem. I have not decided yet which backend to use, it could be an attached storage or amazon glacier.
I red the article http://blogs.splunk.com/2013/03/21/parallel-data-transfer-in-shuttl-0-8-0/ and it seems that installing shuttl on search head you can orchestrate the archiviation in the cluster but I cannot find instructions on how to set this up.
Firstly, I'm unclear about on which nodes have to install shuttl, both search head and indexers?
Secondly, I'm unclear about how setting up shuttl configuration files (archiver.xml/splunk.xml/server.xml) to identify the cluster.
Thirdly, indexes.conf file, distribuited by master node to indexers, must be configured to call the shuttl archiver script with "coldToFrozenScript = $SPLUNK_HOME/etc/apps/shuttl/bin/coldToFrozenScript.sh" for each index that must be archived, as indicated in shuttl documentation? In this case should be shuttl installed also on indexers, but how and who recognize duplicate buckets?
Thanks,
clagese
Thank you for the response. Unfortunately I have some problems.
I configured my cluster to archive data on s3 with shuttl. I installed Shuttl on all nodes. My data are really archived on s3 bucket but there is a problem with archiviation path.
Any path values I set as "archivePath" in archiver.xml(
The result is like a directory without name, in the root of s3 bucket, that contains the directories structure created by shuttle.
For example if I set
In the shuttl log I found these:
2014-04-16 18:56:22,037 INFO com.splunk.shuttl.archiver.archive.BucketFreezer: will="Attempting to archive bucket" index="mytestdb" path="/opt/splunk/var/lib/splunk/mytestdb/colddb/rb_1397667252_1397662059_21_B95021DE-89AB-4A9D-B924-575736C54B81"
2014-04-16 18:56:22,744 INFO com.splunk.shuttl.archiver.archive.BucketFreezer: will="Attempting to archive bucket" index="mytestdb" path="/opt/splunk/var/lib/splunk/mytestdb/db/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82"
2014-04-16 18:56:25,643 INFO com.splunk.shuttl.archiver.archive.recovery.FailedBucketsArchiver: will="Archiving failed buckets" failed buckets="[LocalBucket [getDirectory()=/opt/splunk/shuttl_archiver/data/safe-buckets/mytestdb/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82, getName()=db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82, getIndex()=mytestdb, getFormat()=SPLUNK_BUCKET, getPath()=/opt/splunk/shuttl_archiver/data/safe-buckets/mytestdb/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82, getEarliest()=Wed Apr 16 17:22:40 CEST 2014, getLatest()=Wed Apr 16 18:54:12 CEST 2014, getSize()=15912], LocalBucket [getDirectory()=/opt/splunk/shuttl_archiver/data/safe-buckets/mytestdb/rb_1397667252_1397662059_21_B95021DE-89AB-4A9D-B924-575736C54B81, getName()=rb_1397667252_1397662059_21_B95021DE-89AB-4A9D-B924-575736C54B81, getIndex()=mytestdb, getFormat()=SPLUNK_BUCKET, getPath()=/opt/splunk/shuttl_archiver/data/safe-buckets/mytestdb/rb_1397667252_1397662059_21_B95021DE-89AB-4A9D-B924-575736C54B81, getEarliest()=Wed Apr 16 17:27:39 CEST 2014, getLatest()=Wed Apr 16 18:54:12 CEST 2014, getSize()=16379]]"
2014-04-16 18:56:29,649 INFO com.splunk.shuttl.archiver.archive.BucketShuttlerRunner: will="Archiving bucket" bucket="LocalBucket [getDirectory()=/opt/splunk/shuttl_archiver/data/safe-buckets/mytestdb/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82, getName()=db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82, getIndex()=mytestdb, getFormat()=SPLUNK_BUCKET, getPath()=/opt/splunk/shuttl_archiver/data/safe-buckets/mytestdb/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82, getEarliest()=Wed Apr 16 17:22:40 CEST 2014, getLatest()=Wed Apr 16 18:54:12 CEST 2014, getSize()=15912]"
2014-04-16 18:56:30,700 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath%2Farchive_data%2Fmy_cluster%2FsplunkIndex02%2Fmytestdb%2Fdb_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82%2FCSV' - Unexpected response code 404, expected 200
2014-04-16 18:56:30,701 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath%2Farchive_data%2Fmy_cluster%2FsplunkIndex02%2Fmytestdb%2Fdb_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82%2FCSV' - Received error response with XML message
2014-04-16 18:56:31,089 INFO com.splunk.shuttl.archiver.archive.ArchiveBucketTransferer: will="attempting to transfer bucket to archive" bucket="LocalBucket [getDirectory()=/opt/splunk/shuttl_archiver/data/format-export-dir/mytestdb/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82/SPLUNK_BUCKET/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82, getName()=db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82, getIndex()=mytestdb, getFormat()=CSV, getPath()=/opt/splunk/shuttl_archiver/data/format-export-dir/mytestdb/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82/SPLUNK_BUCKET/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82, getEarliest()=Wed Apr 16 17:22:40 CEST 2014, getLatest()=Wed Apr 16 18:54:12 CEST 2014, getSize()=15912]" destination="/myrootpath/archive_data/my_cluster/splunkIndex02/mytestdb/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82/CSV"
2014-04-16 18:56:31,095 INFO com.splunk.shuttl.archiver.filesystem.transaction.TransactionExecuter: will="Prepare transaction" transaction="Transaction [data=LocalBucket [getDirectory()=/opt/splunk/shuttl_archiver/data/format-export-dir/mytestdb/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82/SPLUNK_BUCKET/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82, getName()=db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82, getIndex()=mytestdb, getFormat()=CSV, getPath()=/opt/splunk/shuttl_archiver/data/format-export-dir/mytestdb/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82/SPLUNK_BUCKET/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82, getEarliest()=Wed Apr 16 17:22:40 CEST 2014, getLatest()=Wed Apr 16 18:54:12 CEST 2014, getSize()=15912], remoteTemp=/myrootpath/temporary_data/splunkIndex02/myrootpath/archive_data/my_cluster/splunkIndex02/mytestdb/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82/CSV, dst=/myrootpath/archive_data/my_cluster/splunkIndex02/mytestdb/db_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82/CSV]"
2014-04-16 18:56:31,230 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath%2Farchive_data%2Fmy_cluster%2FsplunkIndex02%2Fmytestdb%2Fdb_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82%2FCSV' - Unexpected response code 404, expected 200
2014-04-16 18:56:31,230 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath%2Farchive_data%2Fmy_cluster%2FsplunkIndex02%2Fmytestdb%2Fdb_1397667252_1397661760_22_B95021DE-89AB-4A9D-B924-575736C54B82%2FCSV' - Received error response with XML message
2014-04-16 18:56:31,608 INFO com.splunk.shuttl.archiver.archive.BucketShuttlerRunner: will="Archiving bucket" bucket="LocalBucket [getDirectory()=/opt/splunk/shuttl_archiver/data/safe-buckets/mytestdb/rb_1397667252_1397662059_21_B95021DE-89AB-4A9D-B924-575736C54B81, getName()=db_1397667252_1397662059_21_B95021DE-89AB-4A9D-B924-575736C54B81, getIndex()=mytestdb, getFormat()=SPLUNK_BUCKET, getPath()=/opt/splunk/shuttl_archiver/data/safe-buckets/mytestdb/rb_1397667252_1397662059_21_B95021DE-89AB-4A9D-B924-575736C54B81, getEarliest()=Wed Apr 16 17:27:39 CEST 2014, getLatest()=Wed Apr 16 18:54:12 CEST 2014, getSize()=16379]"
2014-04-16 18:56:31,741 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath' - Unexpected response code 404, expected 200
2014-04-16 18:56:31,741 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath' - Received error response with XML message
2014-04-16 18:56:31,988 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath%2Farchive_data%2Fmy_cluster%2FsplunkIndex01%2Fmytestdb%2Fdb_1397667252_1397662059_21_B95021DE-89AB-4A9D-B924-575736C54B81%2FCSV' - Unexpected response code 404, expected 200
2014-04-16 18:56:31,989 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath%2Farchive_data%2Fmy_cluster%2FsplunkIndex01%2Fmytestdb%2Fdb_1397667252_1397662059_21_B95021DE-89AB-4A9D-B924-575736C54B81%2FCSV' - Received error response with XML message
2014-04-16 18:56:32,024 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath%2Ftemporary_data' - Unexpected response code 404, expected 200
2014-04-16 18:56:32,024 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/%2Fmyrootpath%2Ftemporary_data' - Received error response with XML message
In spite of this problem the Thaw and flush process of splunk buckets from search head interface works well.
I hope you can give me an answer as soon as possible.
Thanks in advance
Hi Clagese,
Thanks,
- Petter
Shuttl Dev