Hi All, We are embarking on moving our Splunk 8.1.3 servers from old version of RHEL to new RHEL servers. The servers names are going to be different and we currently have the config based on IP/hostname as opposed to alias. We have perused the answers community and as well, referred Splunk documentation and have identified the below approach for indexer cluster migration. There's no single Splunk document that talks about this fairly common occurrence in many environments. We also have raised a Splunk support case for advice but to no avail.  A few details about our existing cluster so you can factor that in while reviewing the below Multisite Indexer cluster with 2 indexers across 2 sites Current RF - 2 SF-2 (Origin 1 and total 2 for both) Daily ingestion of roughly 400-450 GB Current storage of 4 indexers (Roughly 9TB utilised out of 11 TB total hot_warm on each indexer). Roughly 36 TB across the indexer cluster. ~ 20K buckets with approx. 5K buckets in each indexer Retention varies between 3 months, 6 months and some on 1 year It is imperative that we have all data replicated to new servers before we retire the old ones No forwarder-->Indexer discovery Un-clustered search heads(4) including one Splunk ES 6.4.x Key concerns: Storage on existing indexers being overwhelmed. Degraded search performance on cluster Resource util increase on cluster instances Sustained replication Lack of testing ability to test the below at scale. Can you please verify/validate/provide commentary/gotchas on this approach please? Would love suggestions on improving this as well Proposed steps Step 0 - Server Build - Build new CMs and Indexers to existing specs (2RF, 2 SF, copy remote bundles on CM etc.) Step 1 - Cluster master migration Search heads - Add additional cluster master via DS push Place existing CM in maintenance mode/Stop CM Ensure new CM is started and running. Place in maintenance mode Modify server.conf on 1 indexer at a time to point to new CM Disable maintenance mode and allow new cluster master to replicate/bucket fixup Validate and confirm that no errors and no problems are seen. Step 2 - Indexer cluster migration 1 on each site Push changes to forwarders to include 2 new indexers (1 in each site)  Place new CM in maintenance mode. Add 1 indexer in each site with existing SF and RF Remove maintenance mode Issue data rebalance command (Searchable rebalance, end goal of which is to have roughly 3K-3.2K buckets in each of the 6 indexers) Wait for catch up……………………………………………….. Place new CM in maintenance mode. Modify CM RF to 4 from 2. Keep SF as same In the same maintenance mode, modify origin as 3 and total 4 for replication factor. SF remains same (On all 6 indexers.4 old + 2 new). This is to ensure each indexer in a site has 1 copy of each bucket Restart indexer cluster master (and possibly indexers) for new changes to reflect Disable maintenance mode Wait for catch up……………………………………………….. Step 3 - Indexer cluster migration 1 on each site Once caught up and validated, place cluster master in maintenance mode Place one old indexer in offline enforce-counts and add 1 new indexer on each site Disable maintenance mode Immediately issue data rebalance command Wait for catch up……………………………………………….. Step 4 – Cluster migration completion Once validated, place cluster master in maintenance mode Place the 2nd set of old indexers on each site in offline enforce-counts Modify RF to 2 and SF remains the same Modify Origin to 1 and Total to 2 RF on all indexers Restart indexers and CM as appropriate Disable maintenance mode Allow excess bucket fix up activities to complete Once cluster has regained status quo, shutdown old indexers Remove reference to old indexers from forwarders and search heads to avoid warnings/errors   ... View more

Hi All, We have recently upgraded from 7.2.6 to 8.1.3 Splunk and since then, we have been having issues with Sourcefire ingestion from FMC. Splunk and sourcefire version - prior to upgrade - 7.2.6 and 3.0.0 Splunk and sourcefire version - Post upgrade - 8.1.3 and 4.6.0 TA used - https://splunkbase.splunk.com/app/3662/ What we've attempted so far 3.0.0  with compatibility enabled for Python 2.x - Errors out with  Connection reset by peer estreamer.subscriber ERROR    error: \nTraceback (most recent call last):\n  File "/opt/splunk/etc/apps/TA-eStreamer/bin/encore/estreamer/subscriber.py", line 198, in start\n    self.connection.connect()\n  File "/opt/splunk/etc/apps/TA-eStreamer/bin/encore/estreamer/connection.py", line 80, in connect\n    self.socket.connect( ( host, port ) )\n  File "/opt/splunk/lib/python2.7/ssl.py", line 864, in connect\n    self._real_connect(addr, False)\n  File "/opt/splunk/lib/python2.7/ssl.py", line 855, in _real_connect\n    self.do_handshake()\n  File "/opt/splunk/lib/python2.7/ssl.py", line 828, in do_handshake\n    self._sslobj.do_handshake()\nerror: [Errno 104] Connection reset by peer\n 4.6.0 upgraded TA in 8.1.3 - Connection succeeds and collects logs for a while but then, we are met with the errors "Invalid JSON in settings file" followed by Subscriberparser is dead, message - We also found this bug reference, similar to the error -  https://quickview.cloudapps.cisco.com/quickview/bug/CSCvy06369 4.6.0 upgraded TA with compatibility enabled for 2.x - Same as above. Connection succeeds but eventually, stops collection after a while and errors out with the same message as present in 2. Fresh install of 4.6.0 followed by fresh config.  Connects fine to FMC but errors out as below "Error state. Clearing queue" In a nutshell, what used to be a stable stream of logs from FMC is completely broken/fragmented. In all cases, able to use the splencore test to establish successful connection and have restarted the service but no luck. We have been through all articles in community and as well, all suggested troubleshooting but no luck. Any advice on getting this working is much appreciated.  @douglashurd - Can you please advise. Thanks! ... View more