Hi Everyone,
I am currently trying to achieve a quite simple process: set up a scalable way to backup/restore some KVStore collections from production Splunk servers.
Following the appropriate Splunk documentation (https://docs.splunk.com/Documentation/Splunk/7.1.2/Admin/BackupKVstore), i was able to successfully backup my collections in JSON formatted files.
However, when i try to restore them to the same production server, it fails with the following errors (from /opt/splunk/var/log/splunkd.log):
splunk: ERROR 1535113443.183 KVStorageProvider - An error occurred during the last operation ('dropCollection', domain: '5', code: '26'): ns not found
splunk: WARN 1535113443.188 KVStoreAdminHandler - No data found to restore matching specified parameters archiveName="backupfile.tar.gz", appName="all apps", collectionName="collection"
splunk: ERROR 1535113443.188 KVStoreAdminHandler - \n
This seems quite cryptic to me. I am wondering whether anyone could have encounter a similar issue or error message, or could know some troubleshooting tips that will help me solving this?
Just in case someone faces the same issue that i did, here is a fix that Andrew from Splunk Support gave me.
Just edit the following stanza/parameter into your "limits.conf" configuration file:
[kvstore]
max_documents_per_batch_save = <unsigned int>
* The maximum number of documents that can be saved in a single batch
* Default: 1000
After raising this "max_documents_per_batch_save" parameter to a value superior to the number of entries found into your Collection, you should be able to restore it properly.
Just in case someone faces the same issue that i did, here is a fix that Andrew from Splunk Support gave me.
Just edit the following stanza/parameter into your "limits.conf" configuration file:
[kvstore]
max_documents_per_batch_save = <unsigned int>
* The maximum number of documents that can be saved in a single batch
* Default: 1000
After raising this "max_documents_per_batch_save" parameter to a value superior to the number of entries found into your Collection, you should be able to restore it properly.
In addition to the modification above, if the number of entry in you collection exceeds 50000, another "limits.conf" parameter must be modified:
[kvstore]
max_rows_per_query = <unsigned int>
* The maximum number of rows that will be returned for a single query to
a collection.
* If the query returns more rows than the specified value, then returned
result set will contain the number of rows specified in this value.
* Default: 50000
Otherwise, the restored KVstore collection will be truncated to 50000 entries.
Thanks for your reply David. This application seems interesting, but it seems very WebUI-focused, whereas i am trying to use CLI as much as possible, in order to be able to automate things. It is always good to discover new applications though, so i will probably give a go to this app! 🙂
yeah for CLI go for commands added after 7.0.
The app gives you advanced search commands queries only 😄 pretty useful for backing up and importing kv back into SHC.
Yes, i was trying to use those commands, but was hitting a number of entries limit (1000) while using the "restore kvstore" CLI command. A fix was found thanks to Splunk Support as described in my answer below.
It is surely interesting to be able to trigger backup/restore directly from WebUI using SPL.