Getting Data In

migrate data from single splunk indexer, split to two new indexers (sanity check)

jeff
Contributor

I have a plan to migrate data from a single splunk indexer to two separate indexers, reconfiguring the production system from Solaris to RedHat in the process. I've done some testing and it looks like this will work, but need a sanity check. If there are flaws in what I'm proposing let me know... Thanks.

Current environment:

Splunk indexer / web

  • 5TB SAN partition
  • 24 GB RAM
  • Splunk 4.2.5
  • single index (main/defaultdb)
  • Solaris 10 / Intel x64

Phase 1

  1. Bring up second Splunk Indexer
    1. 24GB RAM
    2. Splunk 4.3.1
    3. 3TB SAN partition mounted at /opt/splunk
    4. 3TB SAN partition mounted at /opt/splunktmp
    5. RedHat 6 Enterprise, x64
    6. create "migrate" index in default location, "migratetmp" index in /opt/splunktmp/var/lib/splunk/
  2. copy db_* directories in existing defaultdb ending in an odd number to migrate/colddb, even numbers to migratetmp/colddb:
    rsync -av --progress --stats --rsync-path /opt/sfw/bin/rsync splunk@oldsplunkserver:/opt/splunk/var/lib/splunk/defaultdb/db/db_*{1,3,5,7,9} /opt/splunk/var/lib/splunk/migrate1/colddb/
  3. point all splunk forwarders to new splunk server... no new data to old splunk server
  4. do a final roll of hot to warm on old splunk server, shut down splunk on both servers
  5. do a final rsync to pick up the final bits of log goodness in the old splunk server

Phase 2

  1. Reconfigure old splunk server as a mirror of the new server
  2. detach 3TB partition from the phase1 server mounted at /opt/splunktmp, attach to new server at /opt/splunk
  3. rename migratetmp directory to migrate
  4. configure splunk forwarders to load balance between the two servers

End result should be:

  1. New data coming in in the main/default (or otherwise appropriate index)
  2. Old data available in the migrate index
  3. data will age in the migrate index as defined as time moves on
  4. searches are faster, less storage needed than otherwise (would have to keep an additional 5TB partition for the duration of the data's life), etc.
1 Solution

Brian_Osburn
Builder

Brian_Osburn
Builder

Hrrrm. It sounds overly complicated in my humble opinion, but it should work..

This is what I did:

http://splunk-base.splunk.com/answers/6521/expanding-splunk-installation-from-a-single-indexer-to-a-...

This is a good read as well:
http://docs.splunk.com/Documentation/Splunk/latest/Admin/Moveanindex

jeff
Contributor

Thanks for the confirmation... I had already read the documentation and ran some tests on my own, so I was pretty confident already. My constraint in my situation is the limitation of the two servers. The current production system is going to be refreshed and changed from solaris to redhat, so I don't have the luxury of simply having two servers to move to right off.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...