Why did I get "WARNING: web interface does not seem to be available!" replacing index folders on one machine with index folders from another machine?

[imanpoeiri]

Wanted to give heads up to all Splunkers.

My case was - I tried to import data from one machine to another machine by following steps:
1. Prepare the index folders ( colddb , datamodel_summary, db, thaweddb) from the source machine.
2. Replace current index folder on your target machine with the index folders that you prepared before.
3. Execute splunk start give error said as above.

It was happened to me and I couldn't bring up Splunk enterprise after I used indexes from another machine with steps above. I am not sure why because I could not find any odd errors in splunkd.log, but there is a .dmp file generated. But a WARN log like below appears:

07-23-2015 14:36:14.576 +0800 WARN  IndexProcessor - Indexer was started dirty: splunkd startup may take longer than usual; searches may not be accurate until background fsck completes.

How to recover:
1. Delete that index folder completely.
2. Execute splunk start that will create and give you fresh and empty index.
3. Execute splunk stop
4. Remove the folders ( colddb , datamodel_summary, db, thaweddb)
5. Drop the folders from source machine.

[alacercogitatus · Accepted solution]

This is not a question. Also - don't do that. While technically possible, it isn't recommend for a variety of reasons including bucket collision.

If you would like to still copy the information from one system to another, Leave the index folders alone. Copy only the "buckets" located in the colddb. Only copy the warm buckets from the db folder. Copying hot will most likely leave you with corrupted data. You shouldn't need thaweddb at all, since you aren't bringing over frozen buckets.

OF course, things get even more complicated in a Clustered environment. So be wary of what you are doing. Tread lightly, and carry a big bucket.

You may want to checkout this doc on buckets:
http://docs.splunk.com/Documentation/Splunk/6.2.4/Indexer/HowSplunkstoresindexes

[imanpoeiri]

Thanks for the advise @alacercogitatus! That carved in my mind now.

The reason why I post it because there would be other beginner splunker like me out there - they know what they want to achieve but may not understand what they are doing with the indexes and ended up with cant bring the server up.

And yea, it is only for testing perusal and I wont do it on the live environment.

Thanks and cheers once again!