Proper Splunk Indexer Replacement Procedure

[bloom_dfarrell]

I'm working on a procedure to move from an old indexer to a new indexer without losing any events. The configuration is pretty simple: there are a number of Universal Forwarders which send to one indexer, and we'd like to replace it with another indexer. Eventually the old indexer will be disabled. Everything's basically vanilla.

Questions

My goal asking this question is to tap into the knowledge base of the splunk community to determine:

1. Sanity check. Is this approach reasonable? Is there a better approach?
2. Can I really replicate between indexers by forwarding like this?
   
3. Am I correct in thinking the Universal Forwarders won't cause any problems by blocking?
   
4. Any further input is much appreciated.

Requirements:

• no lost events
• don't want to have to change configuration on the live forwarders on the fly, but rather would like the process to be more or less invisible to them, besides a DNS change.
• a bit of splunk downtime is acceptable.

Procedure

I thought the best procedure would be something like this:

1. spin up new server "splunk2". It will already contain the same configuration as the old server.
2. Shut down old server "splunk1". Universal Forwarders begin to queue and might eventually block. But, since they forward only locally, this won't cause any cascading blocking (dropEventsOnQueueFull is set to the default of -1)
3. Roll hot indexes to warm. I think maybe this will happen automatically but I'm unclear as to whether it's at startup or shutdown.
4. Copy all indexes from splunk1 to splunk2. Since the hot indexes have already been rolled this would only include warm and below. Now the servers contain the exact same indexes.
   
5. Add the new server splunk2 as an Output on the old server splunk1.
   
6. Start splunk2 and make sure it comes up with the indexes moved over.
   
7. Start splunk1. It should quickly receive all the queued events from the forwarders and forward those events to splunk2 as well as indexing them locally.
8. Start searching again on splunk2. Do Sanity checks. Up to this point, the procedure can be rolled back by simply removing the forward on splunk1 and discarding splunk2.
9. Update DNS to point to splunk2. As the change propagates to the forwarders they will cut splunk1 out of the loop and their events will be sent directly to splunk2. However since any old server events will still be forwarded, nothing will be lost. This the trigger pull step, as after this is done there's no easy way to rollback (I suppose the indexes would have to be merged or something)
10. Wait for the TTL of the DNS record to expire, + dnsResolutionInterval, for the changes to be guaranteed to propagate. After the TTL expires all records on intermediary DNS servers should expire, and dnsResolutionInterval

Thanks for any advice you care to share!

[Simeon · Accepted solution]

The outline you have is a good start. Here is how I do it at a high level:

1. Turn on new indexer (enable distributed search to other indexer)
2. Move inputs to send to new indexer
3. Slowly phase out old indexer

This route provides for roll-back in case you mess up on part 2. For that step, you typically need to update the forwarders with the new indexer IP. In some deployments, it's as easy as updating DNS to go to the other host. It really depends on your deployment. Using the distributed search functionality will allow you to slowly phase out the old indexer without having to copy over data immediately. With the latest licensing server setup, you might need to setup the new indexer as the master license server. Once you find that most of the current data is in the new indexer, you can begin your copying of the old data. I recommend you copy the old data to newly named indexes that have a prefix like "old_" (or just rename them old_*). This way, there is no bucket collision when you put this data on the new indexer.

[bloom_dfarrell]

Thanks, your input is much appreciated!