Multi-site cluster , wrong Restrictive retention caused data delete. Anyway to stop splunk from deleting rest of teh data.

[sat94541]

Issue: Cluster Master with multi-site, by mistake wrong retention file was replicated and for the entire indexer in site1 Data got deleted.
Now only peer in site2 has data.As soon as we realized this mistake we have shut down the splunk instance.
What steps can we take to save as much as data possible.

[scheng_splunk]

detailed steps also posted in https://answers.splunk.com/answers/559610/wrong-retention-caused-buckets-to-freeze-in-cluste.html?ch...

[rbal_splunk]

Shutdown all of the indexers and modify indexes.conf and change the retention back to higher value.

Assuming mistake was only on one peer (or one site), you'll lose buckets on that peer. The other buckets will still be around, but marked as frozen.
You need to take the folloing steps to unmark these buckets from “frozen” to “non-frozen”

i)Get a list of frozen buckets using curl commands like

https:// :8089/services/cluster/master/buckets?filter=frozen=true
https://:8089/services/cluster/master/buckets?filter=frozen=true&filter=index=

OR, from the CM you can run

| rest services/cluster/master/buckets?filter=frozen=true&filter=index=ABCD

NOTE: Such rest call from CM can be used as long as there’s not a million buckets that are frozen

ii)Now for each of these bucket edit “bucket_info” and flip the frozen flag from 1 to “”

For example for bucket $SPLUNK_HOME/var/lib/splunk//db/db_1476118503_1476117359_6_6DC24692-777A-424E-AE9D-5FD23FAC185B/ bucket_info

or from the CM you can run

| rest services/cluster/master/buckets?filter=frozen=true&filter=index=ABCD

NOTE: Such rest call from CM can be used as long as there’s not a million buckets that are frozen

Change from>

"indextime_et","indextime_lt","frozen_in_cluster"
1475514557,"",1

change to>
"indextime_et","indextime_lt","frozen_in_cluster"
1475514557,"",""

Also for index, delete $SPLUNK_HOME/var/lib/splunk//db/.bucketManifest
After for all the bucket_info is fixed for all index and indexer, you may restart the indexers.