Upgrade Splunk to newer version

[amitm05]

I have a Splunk Enterprise Clustered environment and I've TBs of data coming in per day.
Now, while going for an upgrade of my splunk on Indexers and Search Heads - I want to talk about and clear my doubt about my indexed data backup (especially the hot and warm buckets).

1. What would the best practice. Whether to stop all the indexers and upgrade them and then start them ? Although I feel this will pose a downtime and will increase to the choking of forwarders when the indexers come back online.
   OR
   I should go for one by one upgrade of the indexers. In this approach after the I start upgrading the indexers and while its in progress, the old versioned and new versioned Indexers will have to work in sync. Does that cause any problem ?

2. After upgrading the indexer and restart- Do the hot bucket resumes seamlessly ?

Please do not just provide http://docs.splunk.com/Documentation/Splunk/6.4.0/Indexer/Backupindexeddata

Thanks !

[FrankVl · Accepted solution]

Instead of looking at the backup documentation, I would suggest to take a look at the upgrade documentation:
http://docs.splunk.com/Documentation/Splunk/latest/Installation/UpgradeyourdistributedSplunkEnterpri...

(Make sure to select the relevant Splunk version, I linked to the latest version documentation)

I think in general a one by one upgrade would make more sense, as taking the entire indexer cluster offline is bound to lead to data loss (unless your type of data sources and forwarder architecture has sufficient caching capability in it to manage such an extended downtime of all indexers).

[xpac]

From what I've seen, starting from 7.1.0. rolling upgrades are supported 🙂

[amitm05]

What are rolling upgrades, how do they work ?

[FrankVl]

See: http://docs.splunk.com/Documentation/Splunk/7.1.0/DistSearch/SHCrollingupgrade and http://docs.splunk.com/Documentation/Splunk/7.1.0/Indexer/Searchablerollingupgrade

But as mentioned: that feature is new in 7.1.0, so only becomes useful when upgrading from 7.1.0 to a future version.

[FrankVl]

Yeah, that sounds really nice. You'd have to get to 7.1.0 first though, so I guess not too relevant for the @amitm05

[xpac]

Yep, just thought that would be a good reason to consider which version of Splunk to update to. 😉