I understand that this is an older question, but feel it should be answered anyway.
You can run the snapshots while the SPLUNK systems are running. You can stop the SPLUNK systems to do snapshots as well to make sure all data is copied at that point in time. The only reason to stop SPLUNK to do snapshots, is to collect the "hot" or actively transacted data (actually the hot bucket data cannot be backed up even with a traditional backup program, the data is simply written to the warm bucket when the system is stopped, as warm and cold buckets are the only ones backed up). Most people do not have much data in the hot buckets and are usually not critical to a restoration of the system and data integrity from a restoration point of view. However, if you are paranoid or very critical of your data, then you can stop SPLUNK and do the snapshots. Remember to not restart SPLUNK until all snapshots are done. Otherwise your data may be mismatched as SPLUNK data is spread across the indexers in the system, as no single indexer (unless you only have one indexer) contains all the data in the SPLUNK system. While most other people to save time and resources, running the snapshots while the SPLUNK system is still up is just fine.
... View more