I am in a situation where I need to migrate my the splunk indexes to a bigger drive. I was wondering what would be a good way of accomplishing this.
I guess my question is what is the best way to shutdown these servers and upgrade them one at a time? Will this cause any issues? What happens with The Universal Forwarders?
My environment consists of 2 Search heads and 2 indexers with several Universal forwarders sending logs.
Queue whatever the forwarder would hold, which is normally only a few hundred or thousand events, then the forwarders would stop accepting data.
What would happen if both indexers were down. Does the data just queue? Or does it gets lost and I will miss all data coming in from the UF's? If so what is the best way to make sure I don't lose any new data?
If you are sending from Splunk forwarders, then yes, while one indexer is down, all new data will go to the remaining ones. Assuming that one indexer can handle the load, the downside of this is really just that your data for that period will be unbalanced, so if you search for any data collected during the time, it's all stored on one node, so one node does all the work of retrieving the data. Over time, both will balance out, and if it's for a relatively short period (a few hours) there is no long-term harm. Of course the other disadvantages would be that if your one remaining server stopped while you were upgrading, you would of course be unable to index at all, and during the upgrade, data on the down indexer will be unavailable (and so searches will return incomplete results) but that I think is and obvious consequence.
What would happen if both indexers were down. Does the data just queue? Or does it gets lost and I will miss all data coming in from the UF's? If so what is the best way to make sure I don't lose any new data?