Merging two databases

[yonilevy]

Hi,

I know there are a couple of similar questions over here, but none of them were specific enough for me to be sure of whether what i want to do is possible and how to do it.

I've had a small fuckup on my splunk instance, where the machine had restarted without an up-to-date mount setup in /etc/fstab, resulting in SPLUNK_DB being configured to a non-existant path when the machine came back up. What I believe happened next is splunk created the database directories over there, basically starting from scratch (off-topic, but IMO a better behaviour would be to warn the SPLUNK_DB path pointing to an empty directory in UI and let the user choose to create a new database).

Now, I have the old database somewhere on disk, and the new database which has been collecting logs for a few days, I'd like to merge the two so that all my log data is accessible through splunk.

Listing the splunk/defaultdb/db directories outputs this:

New Database:

-rw------- 1 splunk root    10 2014-09-17 13:12 CreationTime
drwx--x--x 3 splunk root  4096 2014-09-18 17:10 db_1411041667_1410937646_0
drwx--x--x 3 splunk root  4096 2014-09-18 17:10 db_1411049354_1411041668_1
drwx--x--x 3 splunk root  4096 2014-09-18 17:21 db_1411049994_1411038557_2
drwx--x--x 3 splunk root  4096 2014-09-21 16:46 db_1411132659_1411039187_3
drwx--x--x 2 splunk root  4096 2014-09-17 13:12 GlobalMetaData
-rw------- 1 splunk root   710 2014-09-21 18:43 Hosts.data
drwx--x--x 3 root   root 12288 2014-09-21 18:44 hot_v1_4
-rw------- 1 root   root 20821 2014-09-21 18:43 Sources.data
-rw------- 1 splunk root   116 2014-09-21 18:43 SourceTypes.data

Old Database:

-rw------- 1 splunk splunk     10 2010-12-09 12:31 CreationTime
drwx--x--x 3 root   root    12288 2014-04-13 17:16 db_1397300990_1397107062_163
drwx--x--x 3 root   root     4096 2014-04-29 10:04 db_1397429712_1397429712_173
drwx--x--x 3 root   root    12288 2014-04-16 08:00 db_1397516218_1397300991_164
drwx--x--x 3 root   root    12288 2014-04-18 16:00 db_1397725464_1397516219_165
drwx--x--x 3 root   root     4096 2014-04-25 12:01 db_1397753810_1397429712_171
drwx--x--x 3 root   root     4096 2014-05-03 04:04 db_1397753810_1397753741_176
drwx--x--x 3 root   root    12288 2014-04-20 20:00 db_1397906465_1397725465_166
drwx--x--x 3 root   root    12288 2014-04-22 20:00 db_1398083195_1397906466_168
drwx--x--x 3 root   root    12288 2014-04-24 20:00 db_1398259989_1398083196_169
drwx--x--x 3 root   root    12288 2014-04-27 04:01 db_1398456185_1398259990_170
drwx--x--x 3 root   root    12288 2014-04-29 20:00 db_1398679459_1398456186_172
drwx--x--x 3 root   root    12288 2014-05-02 00:00 db_1398869854_1398679460_174
drwx--x--x 3 root   root    12288 2014-05-04 08:00 db_1399071488_1398869855_175
drwx--x--x 3 root   root    12288 2014-05-06 16:00 db_1399281831_1399071489_177
drwx--x--x 3 root   root    12288 2014-05-08 20:02 db_1399471134_1399281832_178
drwx--x--x 3 root   root    12288 2014-05-11 04:00 db_1399660803_1399471135_179
drwx--x--x 3 root   root    12288 2014-05-13 13:35 db_1399878091_1399660804_180
drwx--x--x 3 root   root    12288 2014-05-15 14:37 db_1400054520_1399878092_181
drwx--x--x 3 root   root     4096 2014-07-15 10:57 db_1400077085_1400034025_224
drwx--x--x 3 root   root    12288 2014-05-17 09:00 db_1400206910_1400054521_182
drwx--x--x 3 root   root    12288 2014-05-19 08:00 db_1400376389_1400206911_184
drwx--x--x 3 root   root     4096 2014-06-04 12:05 db_1400456559_1400456559_196
drwx--x--x 3 root   root     4096 2014-05-26 11:16 db_1400538720_1400538720_190
drwx--x--x 3 root   root    12288 2014-05-21 16:00 db_1400572697_1400376390_185
drwx--x--x 3 root   root    12288 2014-05-23 20:00 db_1400731673_1400456559_187
drwx--x--x 3 root   root    12288 2014-05-25 04:00 db_1400870456_1400731674_188
drwx--x--x 3 root   root    12288 2014-05-27 18:08 db_1401028394_1400456559_189
drwx--x--x 3 root   root    12288 2014-05-29 04:00 db_1401225385_1401028395_191
drwx--x--x 3 root   root    12288 2014-05-31 00:00 db_1401383945_1401225386_192
drwx--x--x 3 root   root    12288 2014-06-01 20:00 db_1401544683_1401383946_193
drwx--x--x 3 root   root    12288 2014-06-03 20:00 db_1401713431_1401544684_194
drwx--x--x 3 root   root    12288 2014-06-05 18:03 db_1401876975_1401713432_195
drwx--x--x 3 root   root    12288 2014-06-07 16:00 db_1402041708_1401876976_197
drwx--x--x 3 root   root    12288 2014-06-09 12:00 db_1402203933_1402041709_198
drwx--x--x 3 root   root    12288 2014-06-11 04:01 db_1402349799_1402203934_199
drwx--x--x 3 root   root    12288 2014-06-13 12:00 db_1402493163_1402349800_200
drwx--x--x 3 root   root     4096 2014-06-20 04:03 db_1402506723_1402506723_206
drwx--x--x 3 root   root    12288 2014-06-14 20:01 db_1402654522_1402493164_201
drwx--x--x 3 root   root    12288 2014-06-16 16:01 db_1402819967_1402654523_202
drwx--x--x 3 root   root     4096 2014-06-27 04:07 db_1402930370_1402930370_211
drwx--x--x 3 root   root    12288 2014-06-19 11:02 db_1402953327_1402819968_203
drwx--x--x 3 root   root     4096 2014-07-02 00:05 db_1403007900_1402983000_215
drwx--x--x 3 root   root    12288 2014-06-20 00:01 db_1403105477_1402953328_204
drwx--x--x 3 root   root    12288 2014-06-21 20:00 db_1403264454_1403105478_205
drwx--x--x 3 root   root    12288 2014-06-23 00:01 db_1403369507_1403264455_207
drwx--x--x 3 root   root    12288 2014-06-24 20:01 db_1403531642_1403369508_208
drwx--x--x 3 root   root    12288 2014-06-26 17:13 db_1403694458_1403531643_209
drwx--x--x 3 root   root    12288 2014-06-28 08:01 db_1403829507_1403694459_210
drwx--x--x 3 root   root    12288 2014-06-30 00:01 db_1403964393_1403829508_212
drwx--x--x 3 root   root    12288 2014-07-01 18:02 db_1404129436_1403964394_213
drwx--x--x 3 root   root    12288 2014-07-03 11:46 db_1404285251_1404114300_214
drwx--x--x 3 root   root    12288 2014-07-04 20:05 db_1404406951_1404285252_216
drwx--x--x 3 root   root    12288 2014-07-06 10:56 db_1404546898_1404406952_217
drwx--x--x 3 root   root    12288 2014-07-07 20:29 db_1404667691_1404546899_218
drwx--x--x 3 root   root    12288 2014-07-10 00:00 db_1404833766_1404667692_219
drwx--x--x 3 root   root    12288 2014-07-11 01:16 db_1404933208_1404833767_220
drwx--x--x 3 root   root     4096 2014-07-24 12:49 db_1404957367_1404957367_232
drwx--x--x 3 root   root    12288 2014-07-12 16:55 db_1405075538_1404933209_221
drwx--x--x 3 root   root    12288 2014-07-14 04:00 db_1405191793_1405075539_222
drwx--x--x 3 root   root    12288 2014-07-15 15:29 db_1405327657_1405191794_223
drwx--x--x 3 root   root    12288 2014-07-17 04:00 db_1405454792_1405327658_225
drwx--x--x 3 root   root    12288 2014-07-18 08:01 db_1405561398_1405454793_226
drwx--x--x 3 root   root    12288 2014-07-20 00:00 db_1405694015_1405561399_227
drwx--x--x 3 root   root    12288 2014-07-21 10:30 db_1405826759_1405694016_228
drwx--x--x 3 root   root    12288 2014-07-23 11:18 db_1405942873_1405826760_229
drwx--x--x 3 root   root    12288 2014-07-24 14:02 db_1406101489_1405942874_230
drwx--x--x 3 root   root    12288 2014-07-25 20:01 db_1406198559_1406101490_231
drwx--x--x 3 root   root    12288 2014-07-27 04:01 db_1406323904_1406198560_233
drwx--x--x 3 root   root    12288 2014-07-28 20:01 db_1406461547_1406303520_234
drwx--x--x 3 root   root    12288 2014-07-30 12:00 db_1406590970_1406461548_235
drwx--x--x 3 root   root    12288 2014-07-31 20:00 db_1406718419_1406590971_236
drwx--x--x 3 root   root    12288 2014-08-02 00:01 db_1406828051_1406718420_237
drwx--x--x 3 root   root    12288 2014-08-04 11:54 db_1406951768_1406735160_238
drwx--x--x 3 root   root    12288 2014-08-05 11:49 db_1407076589_1406951769_239
drwx--x--x 3 root   root     4096 2014-08-08 12:02 db_1407123960_1407123960_243
drwx--x--x 3 root   root    12288 2014-08-06 16:00 db_1407229897_1407076590_240
drwx--x--x 3 root   root    12288 2014-08-08 20:00 db_1407329841_1407216000_241
drwx--x--x 3 root   root     4096 2014-08-13 00:01 db_1407420660_1407158580_247
drwx--x--x 3 root   root    12288 2014-08-09 12:00 db_1407439593_1407329842_242
drwx--x--x 3 root   root    12288 2014-08-10 15:27 db_1407568932_1407439594_244
drwx--x--x 3 root   root    12288 2014-08-12 00:01 db_1407688866_1407568933_245
drwx--x--x 3 root   root    12288 2014-08-13 14:12 db_1407829796_1407688867_246
drwx--x--x 3 root   root     4096 2014-08-18 14:56 db_1407904560_1407904560_252
drwx--x--x 3 root   root    12288 2014-08-15 00:00 db_1407938868_1407829797_248
drwx--x--x 3 root   root    12288 2014-08-16 04:01 db_1408044586_1407938869_249
drwx--x--x 3 root   root     4096 2014-08-24 04:04 db_1408114425_1408114425_258
drwx--x--x 3 root   root    12288 2014-08-17 16:01 db_1408176730_1408044587_250
drwx--x--x 3 root   root    12288 2014-08-19 00:01 db_1408292591_1408176731_251
drwx--x--x 3 root   root    12288 2014-08-20 11:37 db_1408424369_1408292592_253
drwx--x--x 3 root   root     4096 2014-08-25 14:30 db_1408539420_1408539420_260
drwx--x--x 3 root   root    12288 2014-08-21 20:01 db_1408543150_1408424370_254
drwx--x--x 3 root   root    12288 2014-08-23 00:00 db_1408641011_1408543151_255
drwx--x--x 3 root   root    12288 2014-08-24 08:00 db_1408745528_1408641012_256
drwx--x--x 3 root   root    12288 2014-08-25 14:30 db_1408865099_1408745529_257
drwx--x--x 3 root   root    12288 2014-08-26 20:00 db_1408973060_1408865100_259
drwx--x--x 3 root   root    12288 2014-08-28 11:17 db_1409105264_1408973061_261
drwx--x--x 3 root   root    12288 2014-08-30 03:41 db_1409213675_1409105265_262
drwx--x--x 3 root   root    12288 2014-08-30 14:37 db_1409312139_1409213676_263
drwx--x--x 3 root   root     4096 2014-09-03 20:01 db_1409358619_1409358619_269
drwx--x--x 3 root   root    12288 2014-09-01 00:00 db_1409410906_1409312140_265
drwx--x--x 3 root   root    12288 2014-09-02 08:00 db_1409526157_1409410907_266
drwx--x--x 3 root   root    12288 2014-09-04 00:00 db_1409667148_1409526158_267
drwx--x--x 3 root   root    12288 2014-09-05 04:01 db_1409778727_1409667149_268
drwx--x--x 3 root   root    12288 2014-09-06 12:00 db_1409895259_1409778728_270
drwx--x--x 3 root   root    12288 2014-09-07 11:54 db_1409993602_1409895260_271
drwx--x--x 3 root   root    12288 2014-09-08 19:03 db_1410105608_1409993603_272
drwx--x--x 3 root   root    12288 2014-09-10 15:56 db_1410253485_1410105609_273
drwx--x--x 3 root   root     4096 2014-09-11 20:01 db_1410353779_1410253486_274
drwx--x--x 3 root   root    12288 2014-09-13 04:00 db_1410462285_1410353780_275
drwx--x--x 3 root   root    12288 2014-09-14 08:01 db_1410570138_1410462286_276
drwx--x--x 3 root   root    12288 2014-09-15 12:17 db_1410686173_1410570139_277
drwx--x--x 3 root   root    12288 2014-09-17 04:00 db_1410805867_1410686174_278
drwx--x--x 2 splunk splunk   4096 2010-12-09 12:31 GlobalMetaData
-rw------- 1 root   root     1431 2014-09-17 13:07 Hosts.data
-rw------- 1 splunk splunk      0 2010-12-09 14:04 hot_v1_0.sentinel
-rw------- 1 splunk splunk      0 2011-01-18 23:59 hot_v1_1.sentinel
drwx--x--x 3 root   root     4096 2014-09-17 13:06 hot_v1_279
drwx--x--x 3 root   root     4096 2014-09-17 13:07 hot_v1_280
-rw------- 1 splunk splunk      0 2011-01-21 16:22 hot_v1_2.sentinel
-rw------- 1 splunk splunk      0 2011-01-30 10:01 hot_v1_3.sentinel
-rw------- 1 root   root   776742 2014-09-17 13:07 Sources.data
-rw------- 1 root   root      122 2014-09-17 13:07 SourceTypes.data

So it looks like there's no conflicts in those data "buckets" or however they're called.

How should I go about merging the two databases?
Any more information I should provide?

Thanks ahead!

[datasearchninja]

I'm sure this is not a Splunk recommended practice, but is reasonably easy to achieve so long as you are not using index clustering.

In your case, because the bucket ID's from the instance that has been running for a few days, are all lower than, and don't exist in the older index, you should be able to just:

1. Force a roll of hot to warm buckets
2. Stop splunk
3. Move the db_* folders from the new to the old location
4. Get the old location mounted in the correct location
5. Start splunk

See MoveIndexes

With index clustering/replication it is more difficult to determine if duplicate buckets are present.

[gkanapathy]

This is actually fine per Splunk practice. A couple of details though:

• You don't need to force the roll to warm, though it's marginally easier if you do.
• You can ignore/delete the .sentinel files
• Make sure there are no bucket ID conflicts. The bucket ID is the last number in the bucket directory name (999 in db_1234567890_2345678901_999, or in hot_v1_999 if you didn't roll buckets.). If you have a conflict, simply rename the directory to a number that does not conflict when you copy it. The number does not have to be consecutive, there can be gaps. In your case there are no conflicts, as your old db no longer has ID's 0 through 3 (But you should check the colddb folder to make sure. If they're there, rename 0 thru 3 to, e.g., 290 thru 293.)

[yonilevy]

Thank you both, I've managed to successfully merge the two database using the method described. However, I found a third(!) database on disk, and when trying to merge that one in using the same method, after starting splunk all the copied db directories (db_xxxxxxx_yyy) got deleted. For some reason it won't take them in. Any ideas?

[yonilevy]

Thanks, I'll try that and update here.