Getting Data In

Shutting down splunk Indexers For Upgrade

paul_1994
Path Finder

I am in a situation where I need to migrate my the splunk indexes to a bigger drive. I was wondering what would be a good way of accomplishing this.

I guess my question is what is the best way to shutdown these servers and upgrade them one at a time? Will this cause any issues? What happens with The Universal Forwarders?

My environment consists of 2 Search heads and 2 indexers with several Universal forwarders sending logs.

  1. my concern is what happens when I shutdown an Indexer.Does all the new data just go to one Indexer?
  2. When upgrading each server is there a problem having this Server down for 2-3 hours?
0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Queue whatever the forwarder would hold, which is normally only a few hundred or thousand events, then the forwarders would stop accepting data.

paul_1994
Path Finder

What would happen if both indexers were down. Does the data just queue? Or does it gets lost and I will miss all data coming in from the UF's? If so what is the best way to make sure I don't lose any new data?

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

If you are sending from Splunk forwarders, then yes, while one indexer is down, all new data will go to the remaining ones. Assuming that one indexer can handle the load, the downside of this is really just that your data for that period will be unbalanced, so if you search for any data collected during the time, it's all stored on one node, so one node does all the work of retrieving the data. Over time, both will balance out, and if it's for a relatively short period (a few hours) there is no long-term harm. Of course the other disadvantages would be that if your one remaining server stopped while you were upgrading, you would of course be unable to index at all, and during the upgrade, data on the down indexer will be unavailable (and so searches will return incomplete results) but that I think is and obvious consequence.

paul_1994
Path Finder

What would happen if both indexers were down. Does the data just queue? Or does it gets lost and I will miss all data coming in from the UF's? If so what is the best way to make sure I don't lose any new data?

0 Karma
Get Updates on the Splunk Community!

Observability | Use Synthetic Monitoring for Website Metadata Verification

If you are on Splunk Observability Cloud, you may already have Synthetic Monitoringin your observability ...

More Ways To Control Your Costs With Archived Metrics | Register for Tech Talk

Tuesday, May 14, 2024  |  11AM PT / 2PM ET Register to Attend Join us for this Tech Talk and learn how to ...

.conf24 | Personalize your .conf experience with Learning Paths!

Personalize your .conf24 Experience Learning paths allow you to level up your skill sets and dive deeper ...