How to move data from an index in one environment to an index in a new environment?

[brent_weaver]

I need to move index data from one environment to another while [hopefully] consolidating them into fewer indexes. Is there a way to extract the data and then in turn import it into whatever index I want? Will the metadata case as issue? What are my options?

[lguinn2]

There is no good way to consolidate indexes. Here is what will be clean and work well, but will take time

Assume original indexes are A and B.
Before you begin, plan everything. You will need to take both your Splunk environments (old and new) offline while you copy data between them, to avoid missed events and/or corruption.

To move A and B to the new environment, first you need to find where they are defined in indexes.conf on the old environment. Make sure you copy all of the directories from the proper old locations to the proper new locations. Do not consolidate any directories during the copy. (if you do, it will probably corrupt your indexes...)

Edit indexes.conf in the new environment to match the new location of the indexes.
Check to make sure that the old indexes are now working properly in the new environment.

Assume that in the new environment, we only want to keep index A. Simply (!) find where inputs are being sent to index B - this will probably occur throughout your Splunk environment, including the forwarders. Switch all the forwarders to the new environment and make sure all the inputs.conf contain only references to index A; there should be no references to index B.

At this point, you can start indexing in the new environment.

Next you need to go edit any searches or roles that refer to index B. In the short term, they will need to refer to both index A and index B. After index B is completely decommissioned, you can remove all references to it from searches and from roles.

At this point, you are no longer putting new data in index B. But you still need to complete the decommissioning process and right now, index B still has data that you may need. Now is the time to set retention on index B, if you haven't already, so that the data in index B will gradually "age out." (frozenTimePeriodInSecs = 7776000 in indexes.conf will set the retention time to 90 days.) After the retention time that you set, index B will be empty and can be completely deleted.

The process that I have described may take several months, but it
1 - avoids the risk of corruption
2 - minimizes downtime (Splunk only has to be down during the actual copying between environments)
3 - avoids doing anything "tricky" that requires deep knowledge and unsupported actions

[esix_splunk]

So there are a few ways you can do this.

1) Export the data in raw and reingest it (painful)
2) Copy the buckets from the indexer to the new indexer (not as painful)

Option 2 is easier assuming :
- Youre working in a non-multisite clustered environment
- Your environment is single instance to single instance

Buckets aren't bound by index names per say, but in clustered environments they are bound by GUIDs and site identifiers. If you're in a single site, you can copy to the new server and restart. Assuming there are no bucket name collisions, Splunk will rebuild the metadata appropriately and they'll be searchable.

Hope that helps.

[hemendralodhi]

How to move into multisite environment. I am facing the issue as indexers are crashing whenever enabling the indexes in indexes.conf file. I copied the data to all 4 indexer ( 2 in each site).

[lguinn2]

@hemendralodhi - @esix is proposing that you copy all the buckets from one indexer to another indexer - not that you make multiple copies of the same bucket across several indexers...