All Apps and Add-ons

"problem parsing indexes.conf [...]" after moving index db

egourmet
Explorer

Hello,
After moving the index db according to http://docs.splunk.com/Documentation/Splunk/latest/Admin/Moveanindex#For_Windows_users: we get the following error message:

Problem parsing indexes.conf: default index disabled - quit!
Validating database failed with code '1'.

We noticed that a few files where not copied over and can't copy them manually.The content of:
- C:\Program Files \ Splunk \ var \ lib \ splunk\persistentstorage
- C:\Program Files \ Splunk \ var \ lib \ splunk \ appserver \ i18n
- C:\Program Files \ Splunk \ var \ lib \ splunk \ appserver \ modules \ static \ css

Any insight much appreciated.

Tags (3)
1 Solution

hexx
Splunk Employee
Splunk Employee

Your default index ("main") is indeed disabled, as advertised by the error message you get on start-up :

system     [main]

(...)

system     defaultDatabase = main
system     disabled = 1  <=====
system     enableRealtimeSearch = true

(...)

Since the disabled = 1 directive comes from $SPLUNK_HOME/etc/system/local/indexes.conf, I would assume that this is caused by a collision in the bucket IDs in that index. In such a situation, Splunk will automatically disable the index on start-up, which is fatal for the default index.

If this is indeed the issue you are encountering, you should be able to fix it by following the instructions listed in this Splunk Answer.

Note that it's quite likely that other indexes which exist by default (_internal, _audit, summary...) may have the same issue, so I would recommend to check those for bucket collisions before re-enabling all disabled indexes in $SPLUNK_HOME/etc/system/local/indexes.conf and attempting to start again.

Let us know if it works!

Update: The general procedure to resolve bucket collisions in a given index goes as follows :

  • stop splunkd
  • determine the lowest available bucket ID in the affected index
  • for each pair of conflicting buckets, pick one of the buckets and change its bucket ID to the next available bucket ID by renaming its directory
  • once you are certain that no bucket ID is shared by two buckets, start splunkd

View solution in original post

hexx
Splunk Employee
Splunk Employee

Your default index ("main") is indeed disabled, as advertised by the error message you get on start-up :

system     [main]

(...)

system     defaultDatabase = main
system     disabled = 1  <=====
system     enableRealtimeSearch = true

(...)

Since the disabled = 1 directive comes from $SPLUNK_HOME/etc/system/local/indexes.conf, I would assume that this is caused by a collision in the bucket IDs in that index. In such a situation, Splunk will automatically disable the index on start-up, which is fatal for the default index.

If this is indeed the issue you are encountering, you should be able to fix it by following the instructions listed in this Splunk Answer.

Note that it's quite likely that other indexes which exist by default (_internal, _audit, summary...) may have the same issue, so I would recommend to check those for bucket collisions before re-enabling all disabled indexes in $SPLUNK_HOME/etc/system/local/indexes.conf and attempting to start again.

Let us know if it works!

Update: The general procedure to resolve bucket collisions in a given index goes as follows :

  • stop splunkd
  • determine the lowest available bucket ID in the affected index
  • for each pair of conflicting buckets, pick one of the buckets and change its bucket ID to the next available bucket ID by renaming its directory
  • once you are certain that no bucket ID is shared by two buckets, start splunkd

hexx
Splunk Employee
Splunk Employee

Glad to know it worked! Feel free to accept my answer to indicate so 🙂

0 Karma

egourmet
Explorer

Indeed, the log shows bucket collision.
02-02-2012 15:32:05.830 -0600 ERROR DatabaseDirectoryManager - Splunk has detected that a directory has been manually copied into its database, causing id conflicts [D:\Splunk\var\lib\splunk\defaultdb\db\db_1308843089_1306506209_14, D:\Splunk\var\lib\splunk\defaultdb\db\db_1310499939_1310495443_14].

However, I am not sure how to rename the conflicting buckets:
D:\Splunk\var\lib\splunk\defaultdb\db\db_1308843089_1306506209_14 dated 1/31/2012
D:\Splunk\var\lib\splunk\defaultdb\db\db_1310499939_1310495443_14 dated 2/1/2012

Thank you again for your help.

egourmet
Explorer

Hexx, thank you for the response. Here is the output:

system [_audit]
system assureUTF8 = false
system blockSignSize = 0
system blockSignatureDatabase = _blocksignature
system coldPath = $SPLUNK_DB\audit\colddb
system coldToFrozenDir =
system coldToFrozenScript =
system compressRawdata = true
system defaultDatabase = main
system disabled = 1
system enableRealtimeSearch = true
system frozenTimePeriodInSecs = 188697600
system homePath = $SPLUNK_DB\audit\db
system indexThreads = auto
system maxConcurrentOptimizes = 3
system maxDataSize = auto
system maxHotBuckets = 3
system maxHotIdleSecs = 0
system maxHotSpanSecs = 7776000
system maxMemMB = 5
system maxMetaEntries = 1000000
system maxRunningProcessGroups = 20
system maxTotalDataSizeMB = 500000
system maxWarmDBCount = 300
system memPoolMB = auto
system minRawFileSyncSecs = disable
system partialServiceMetaPeriod = 0
system quarantineFutureSecs = 2592000
system quarantinePastSecs = 77760000
system rawChunkSizeBytes = 131072
system rotatePeriodInSecs = 60
system serviceMetaPeriod = 25
system suppressBannerList =
system sync = 0
system syncMeta = true
system thawedPath = $SPLUNK_DB\audit\thaweddb
system throttleCheckPeriod = 15
system [_blocksignature]
system assureUTF8 = false
system blockSignSize = 0
system blockSignatureDatabase = _blocksignature
system coldPath = $SPLUNK_DB\blockSignature\colddb
system coldToFrozenDir =
system coldToFrozenScript =
system compressRawdata = true
system defaultDatabase = main
system enableRealtimeSearch = true
system frozenTimePeriodInSecs = 0
system homePath = $SPLUNK_DB\blockSignature\db
system indexThreads = auto
system maxConcurrentOptimizes = 3
system maxDataSize = 1000
system maxHotBuckets = 3
system maxHotIdleSecs = 0
system maxHotSpanSecs = 7776000
system maxMemMB = 5
system maxMetaEntries = 1000000
system maxRunningProcessGroups = 20
system maxTotalDataSizeMB = 0
system maxWarmDBCount = 300
system memPoolMB = auto
system minRawFileSyncSecs = disable
system partialServiceMetaPeriod = 0
system quarantineFutureSecs = 2592000
system quarantinePastSecs = 77760000
system rawChunkSizeBytes = 131072
system rotatePeriodInSecs = 60
system serviceMetaPeriod = 25
system suppressBannerList =
system sync = 0
system syncMeta = true
system thawedPath = $SPLUNK_DB\blockSignature\thaweddb
system throttleCheckPeriod = 15
system [_internal]
system assureUTF8 = false
system blockSignSize = 0
system blockSignatureDatabase = _blocksignature
system coldPath = $SPLUNK_DB\_internaldb\colddb
system coldToFrozenDir =
system coldToFrozenScript =
system compressRawdata = true
system defaultDatabase = main
system enableRealtimeSearch = true
system frozenTimePeriodInSecs = 2419200
system homePath = $SPLUNK_DB\_internaldb\db
system indexThreads = auto
system maxConcurrentOptimizes = 3
system maxDataSize = 100
system maxHotBuckets = 3
system maxHotIdleSecs = 0
system maxHotSpanSecs = 7776000
system maxMemMB = 5
system maxMetaEntries = 1000000
system maxRunningProcessGroups = 20
system maxTotalDataSizeMB = 500000
system maxWarmDBCount = 300
system memPoolMB = auto
system minRawFileSyncSecs = disable
system partialServiceMetaPeriod = 0
system quarantineFutureSecs = 2592000
system quarantinePastSecs = 77760000
system rawChunkSizeBytes = 131072
system rotatePeriodInSecs = 60
system serviceMetaPeriod = 25
system suppressBannerList =
system sync = 0
system syncMeta = true
system thawedPath = $SPLUNK_DB\_internaldb\thaweddb
system throttleCheckPeriod = 15
system [_thefishbucket]
system assureUTF8 = false
system blockSignSize = 0
system blockSignatureDatabase = _blocksignature
system coldPath = $SPLUNK_DB\fishbucket\colddb
system coldToFrozenDir =
system coldToFrozenScript =
system compressRawdata = true
system defaultDatabase = main
system enableRealtimeSearch = true
system frozenTimePeriodInSecs = 2419200
system homePath = $SPLUNK_DB\fishbucket\db
system indexThreads = auto
system maxConcurrentOptimizes = 3
system maxDataSize = 10
system maxHotBuckets = 3
system maxHotIdleSecs = 0
system maxHotSpanSecs = 7776000
system maxMemMB = 5
system maxMetaEntries = 1000000
system maxRunningProcessGroups = 20
system maxTotalDataSizeMB = 500000
system maxWarmDBCount = 300
system memPoolMB = auto
system minRawFileSyncSecs = disable
system partialServiceMetaPeriod = 0
system quarantineFutureSecs = 2592000
system quarantinePastSecs = 77760000
system rawChunkSizeBytes = 131072
system rotatePeriodInSecs = 60
system serviceMetaPeriod = 25
system suppressBannerList =
system sync = 0
system syncMeta = true
system thawedPath = $SPLUNK_DB\fishbucket\thaweddb
system throttleCheckPeriod = 15
system [default]
system assureUTF8 = false
system blockSignSize = 0
system blockSignatureDatabase = _blocksignature
system coldToFrozenDir =
system coldToFrozenScript =
system compressRawdata = true
system defaultDatabase = main
system enableRealtimeSearch = true
system frozenTimePeriodInSecs = 188697600
system indexThreads = auto
system maxConcurrentOptimizes = 3
system maxDataSize = auto
system maxHotBuckets = 3
system maxHotIdleSecs = 0
system maxHotSpanSecs = 7776000
system maxMemMB = 5
system maxMetaEntries = 1000000
system maxRunningProcessGroups = 20
system maxTotalDataSizeMB = 500000
system maxWarmDBCount = 300
system memPoolMB = auto
system minRawFileSyncSecs = disable
system partialServiceMetaPeriod = 0
system quarantineFutureSecs = 2592000
system quarantinePastSecs = 77760000
system rawChunkSizeBytes = 131072
system rotatePeriodInSecs = 60
system serviceMetaPeriod = 25
system suppressBannerList =
system sync = 0
system syncMeta = true
system throttleCheckPeriod = 15
system [history]
system assureUTF8 = false
system blockSignSize = 0
system blockSignatureDatabase = _blocksignature
system coldPath = $SPLUNK_DB\historydb\colddb
system coldToFrozenDir =
system coldToFrozenScript =
system compressRawdata = true
system defaultDatabase = main
system enableRealtimeSearch = true
system frozenTimePeriodInSecs = 604800
system homePath = $SPLUNK_DB\historydb\db
system indexThreads = auto
system maxConcurrentOptimizes = 3
system maxDataSize = 10
system maxHotBuckets = 3
system maxHotIdleSecs = 0
system maxHotSpanSecs = 7776000
system maxMemMB = 5
system maxMetaEntries = 1000000
system maxRunningProcessGroups = 20
system maxTotalDataSizeMB = 500000
system maxWarmDBCount = 300
system memPoolMB = auto
system minRawFileSyncSecs = disable
system partialServiceMetaPeriod = 0
system quarantineFutureSecs = 2592000
system quarantinePastSecs = 77760000
system rawChunkSizeBytes = 131072
system rotatePeriodInSecs = 60
system serviceMetaPeriod = 25
system suppressBannerList =
system sync = 0
system syncMeta = true
system thawedPath = $SPLUNK_DB\historydb\thaweddb
system throttleCheckPeriod = 15
system [main]
system assureUTF8 = false
system blockSignSize = 0
system blockSignatureDatabase = _blocksignature
system coldPath = $SPLUNK_DB\defaultdb\colddb
system coldToFrozenDir =
system coldToFrozenScript =
system compressRawdata = true
system defaultDatabase = main
system disabled = 1
system enableRealtimeSearch = true
system frozenTimePeriodInSecs = 188697600
system homePath = $SPLUNK_DB\defaultdb\db
system indexThreads = auto
system maxConcurrentOptimizes = 6
system maxDataSize = auto_high_volume
system maxHotBuckets = 10
system maxHotIdleSecs = 86400
system maxHotSpanSecs = 7776000
system maxMemMB = 20
system maxMetaEntries = 1000000
system maxRunningProcessGroups = 20
system maxTotalDataSizeMB = 500000
system maxWarmDBCount = 300
system memPoolMB = auto
system minRawFileSyncSecs = disable
system partialServiceMetaPeriod = 0
system quarantineFutureSecs = 2592000
system quarantinePastSecs = 77760000
system rawChunkSizeBytes = 131072
system rotatePeriodInSecs = 60
system serviceMetaPeriod = 25
system suppressBannerList =
system sync = 0
system syncMeta = true
system thawedPath = $SPLUNK_DB\defaultdb\thaweddb
system throttleCheckPeriod = 15
system [splunklogger]
system assureUTF8 = false
system blockSignSize = 0
system blockSignatureDatabase = _blocksignature
system coldToFrozenDir =
system coldToFrozenScript =
system compressRawdata = true
system defaultDatabase = main
system disabled = true
system enableRealtimeSearch = true
system frozenTimePeriodInSecs = 188697600
system indexThreads = auto
system maxConcurrentOptimizes = 3
system maxDataSize = auto
system maxHotBuckets = 3
system maxHotIdleSecs = 0
system maxHotSpanSecs = 7776000
system maxMemMB = 5
system maxMetaEntries = 1000000
system maxRunningProcessGroups = 20
system maxTotalDataSizeMB = 500000
system maxWarmDBCount = 300
system memPoolMB = auto
system minRawFileSyncSecs = disable
system partialServiceMetaPeriod = 0
system quarantineFutureSecs = 2592000
system quarantinePastSecs = 77760000
system rawChunkSizeBytes = 131072
system rotatePeriodInSecs = 60
system serviceMetaPeriod = 25
system suppressBannerList =
system sync = 0
system syncMeta = true
system throttleCheckPeriod = 15
system [summary]
system assureUTF8 = false
system blockSignSize = 0
system blockSignatureDatabase = _blocksignature
system coldPath = $SPLUNK_DB\summarydb\colddb
system coldToFrozenDir =
system coldToFrozenScript =
system compressRawdata = true
system defaultDatabase = main
system enableRealtimeSearch = true
system frozenTimePeriodInSecs = 188697600
system homePath = $SPLUNK_DB\summarydb\db
system indexThreads = auto
system maxConcurrentOptimizes = 3
system maxDataSize = auto
system maxHotBuckets = 3
system maxHotIdleSecs = 0
system maxHotSpanSecs = 7776000
system maxMemMB = 5
system maxMetaEntries = 1000000
system maxRunningProcessGroups = 20
system maxTotalDataSizeMB = 500000
system maxWarmDBCount = 300
system memPoolMB = auto
system minRawFileSyncSecs = disable
system partialServiceMetaPeriod = 0
system quarantineFutureSecs = 2592000
system quarantinePastSecs = 77760000
system rawChunkSizeBytes = 131072
system rotatePeriodInSecs = 60
system serviceMetaPeriod = 25
system suppressBannerList =
system sync = 0
system syncMeta = true
system thawedPath = $SPLUNK_DB\summarydb\thaweddb
system throttleCheckPeriod = 15

hexx
Splunk Employee
Splunk Employee

Could you paste here the output of the following command :
splunk cmd btool indexes list --debug

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...