I'm working on a procedure to move from an old indexer to a new indexer without losing any events. The configuration is pretty simple: there are a number of Universal Forwarders which send to one indexer, and we'd like to replace it with another indexer. Eventually the old indexer will be disabled. Everything's basically vanilla.
Questions
My goal asking this question is to tap into the knowledge base of the splunk community to determine:
Requirements:
Procedure
I thought the best procedure would be something like this:
dropEventsOnQueueFull
is set to the default of -1
)dnsResolutionInterval
, for the changes to be guaranteed to propagate. After the TTL expires all records on intermediary DNS servers should expire, and dnsResolutionInterval
Thanks for any advice you care to share!
The outline you have is a good start. Here is how I do it at a high level:
This route provides for roll-back in case you mess up on part 2. For that step, you typically need to update the forwarders with the new indexer IP. In some deployments, it's as easy as updating DNS to go to the other host. It really depends on your deployment. Using the distributed search functionality will allow you to slowly phase out the old indexer without having to copy over data immediately. With the latest licensing server setup, you might need to setup the new indexer as the master license server. Once you find that most of the current data is in the new indexer, you can begin your copying of the old data. I recommend you copy the old data to newly named indexes that have a prefix like "old_" (or just rename them old_*). This way, there is no bucket collision when you put this data on the new indexer.
The outline you have is a good start. Here is how I do it at a high level:
This route provides for roll-back in case you mess up on part 2. For that step, you typically need to update the forwarders with the new indexer IP. In some deployments, it's as easy as updating DNS to go to the other host. It really depends on your deployment. Using the distributed search functionality will allow you to slowly phase out the old indexer without having to copy over data immediately. With the latest licensing server setup, you might need to setup the new indexer as the master license server. Once you find that most of the current data is in the new indexer, you can begin your copying of the old data. I recommend you copy the old data to newly named indexes that have a prefix like "old_" (or just rename them old_*). This way, there is no bucket collision when you put this data on the new indexer.
Thanks, your input is much appreciated!