Getting Data In

Shutting down splunk Indexers For Upgrade

paul_1994
Path Finder

I am in a situation where I need to migrate my the splunk indexes to a bigger drive. I was wondering what would be a good way of accomplishing this.

I guess my question is what is the best way to shutdown these servers and upgrade them one at a time? Will this cause any issues? What happens with The Universal Forwarders?

My environment consists of 2 Search heads and 2 indexers with several Universal forwarders sending logs.

  1. my concern is what happens when I shutdown an Indexer.Does all the new data just go to one Indexer?
  2. When upgrading each server is there a problem having this Server down for 2-3 hours?
0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Queue whatever the forwarder would hold, which is normally only a few hundred or thousand events, then the forwarders would stop accepting data.

paul_1994
Path Finder

What would happen if both indexers were down. Does the data just queue? Or does it gets lost and I will miss all data coming in from the UF's? If so what is the best way to make sure I don't lose any new data?

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

If you are sending from Splunk forwarders, then yes, while one indexer is down, all new data will go to the remaining ones. Assuming that one indexer can handle the load, the downside of this is really just that your data for that period will be unbalanced, so if you search for any data collected during the time, it's all stored on one node, so one node does all the work of retrieving the data. Over time, both will balance out, and if it's for a relatively short period (a few hours) there is no long-term harm. Of course the other disadvantages would be that if your one remaining server stopped while you were upgrading, you would of course be unable to index at all, and during the upgrade, data on the down indexer will be unavailable (and so searches will return incomplete results) but that I think is and obvious consequence.

paul_1994
Path Finder

What would happen if both indexers were down. Does the data just queue? Or does it gets lost and I will miss all data coming in from the UF's? If so what is the best way to make sure I don't lose any new data?

0 Karma
Get Updates on the Splunk Community!

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...