By data are you referring to _internal and other indices? If you don't already know splunk indices are made of four buckets: hot, warm, cold, frozen. In general you can backup up all of your index's buckets with the exception of the "hot" bucket as splunk is actively writing to the hot bucket.
So what is your back up straegy, incremental or full? If you want to perform full backups with hot buckets you will need to use a snapshot utility. If you do not have the ability or facilities you can manually roll your hot buckets to warm using the command below and backing up your warm buckets and/or other buckets.
splunk _internal call /data/indexes/<index_name>/roll-hot-buckets
If you are using Splunk 5.x you can use cluster repllication as your backup straegy.
As for .conf files which contain all my configutions, I have built mutliple Splunk Apps and Technology Add-ons (TAs) that are deployed by the Deployment Server so config files are not edit locally. My apps are contained in source control and backed up. Alternatively you could backup Deployment Server deploment apps directly and serverclass.conf. Also don't forget to backup your knowledge objects $SPLUNK_HOME\etc\users.
Since I use LDAP authenication and have user roles mapped within an app I dont worry about User and Permissions.
To sum it up in general you should backup the following:
- Indices (always before an upgrade)
- Knowledge objects
- .conf files (custom apps or locally defined)
For restoring I would deploy all my confile first and copy indices to the same directory structure.
additional Reading
WhatisSplunkknowledge
Backupindexeddata
Aboutdeploymentserver
Hope this helps and gets you started, but this is really a can a worms depending on your operational reqirements and mandates. If this helps don't for get to accept and vot it up.
Cheers,
