The KVStore, can be disabled on indexers.
The KVStore should be running on all SH's.
By default splunk has a couple of collections that using KVStore:
[SavedSearchHistory]
type = internal_cache
This is responsible for for keeping track of things like Continuous Scheduled Searches.
Short of that if you are not running a premium app like Enterprise Security or ITSI, most likely you have no other collections in KVStore.
You can tell if you just look for files called collections.conf.
IF you were to use the kvstore for lookups, and the lookups were configured for remote, E.G gonna be done on the peers vs the SH, then in your collections.conf you would configure that collection to replicate = true It would then dump the contents of that collection add it to your search bundle and push it over to the peers and they would use a dumped csv file to perform the remote lookups... meaning you still don't need kvstore running on your indexers.
As far as backing up... There are apps / and processes behind this, but at present we say. Stop splunk / tar it up / start splunk
If you are running SHC, then you will have a primary kvstore member which could be different than the SHC Captain, and then secondaries, they all replicate kvstore data between themselves. If however you have 3 independent SH's they know nothing about each others KVStore.
This is a very high level look... If you are going to be running a premium app that heavily relies on this.... I would suggest more research.
... View more