Deployment Architecture

Splunkd won't restart

MrSplunksta
Path Finder

Hi all!

This windows splunk ver splunk-4.1.2-79191-x86-release running on a windows server 2003 US does not start anymore.

Does anyone know what I should do to fix this problem?

Thanks in advance for your time 🙂

Here's what I know: - the server rebooted by itself following a nessus 4 scan (ouch) - the splunkd service now hangs at each start - splunkd.log reports the following: 06-08-2010 13:29:35.062 ERROR WordPositionData - couldn't parse hash code: - there is a crash log file that contains the following.

[build 79191] C++ exception: object@[0x01CBEB64], type@[0x00D0F58C] Exception is Non-continuable Exception address: [0x77E4BEF7] Crashing thread: indexerPipe ContextFlags: [0x00010007] Dr0: [0x78180475] Dr1: [0x015652AB] Dr2: [0x013C03C1] Dr3: [0x00000001] Dr6: [0x00000014] Dr7: [0x01CBE7F8] SegGs: [0x01CB0000] SegFs: [0x0000003B] SegEs: [0x01CB0023] SegDs: [0x7C820023] Edi: [0x01CBECC8] Esi: [0x01CBEB38] Ebx: [0xFFFFFFFF] Edx: [0x01CBEB64] Ecx: [0x00000000] Eax: [0x01CBEAB0] Ebp: [0x01CBEB00] Eip: [0x77E4BEF7] RaiseException + 60/87 SegCs: [0x011A001B] EFlags: [0x00000206] Esp: [0x01CBEAAC] SegSs: [0x78180023]

OS: Windows Arch: i386

Backtrace: Frame 0 @[0x01CBEB00]: [0x78158E89] CxxThrowException + 70/77 Frame 1 @[0x01CBEB38]: [0x006C64AE] ? Frame 2 @[0x01CBEDFC]: [0x01CBF3A8] ? Frame 3 @[0x00B70610]: (Frame below stack)

Crash dump written to: C:\Program Files\Splunk\var\log\splunk\C__Program Files_Splunk_bin_splunkd_exe_crash-2010-06-08-13-32-30.dmp

ENLIL /5.2 Service Pack 2 C++ Exception type: WordPositionData::Exception -> std::exception what(): couldn't parse hash code: Threads running: 13 terminating...

Tags (1)
0 Karma
1 Solution

MrSplunksta
Path Finder

NOW FIXED!!!

Hi all!

I finally got my issue fixed by issuing the following command before upgrading to 4.1.3

C:\Program Files\Splunk\bin>splunk cmd recover-metadata D:\SplunkDB\SplunkDB\os\ db\hot_v1_2

Hope this helps others!

NOTE: Since my database is in a non default location, you'll have to ajust the command to point to YOUR splunk database location.

View solution in original post

hexx
Splunk Employee
Splunk Employee

The presence of "ERROR WordPositionData - couldn't parse hash code:" messages in splunkd.log often indicates an inconsistency in one of the metadata files (Hosts.data, Sources.data, SourceTypes.data) located in the hot/warm index repository (Example for the main index : $SPLUNK_DB/defaultdb/db/) or in one of the buckets (usually one of the hot ones) contained in that index.

To fix this, the first thing to do is to identify which metadata file(s) has/have inconsistencies.

To that effect, the following command has to be run for the incriminated index (check splunkd.log, it's the index that was just being opened before splunkd crashed) and for all of it's hot/warm buckets :

$SPLUNK_HOME/bin/recover-metadata {path_to_index|path_to_bucket} --validate

For a given index, I like to run the two commands below to check the metadata files at the root of the hot/warm db first, and then each bucket using the list from .bucketManifest :

$SPLUNK_HOME/bin/recover-metadata $SPLUNK_DB/{index_name}/db/ --validate

for i in 'cat $SPLUNK_DB/{index_name}/db/.bucketManifest | cut -f3 -d " "'; do $SPLUNK_HOME/bin/recover-metadata $SPLUNK_DB/{index_name}/db/$i ; done

Each time an error is reported, the corresponding .data file should be moved or deleted. Once all corrupted metadata files have been removed, the check should be run again. It will indicate errors for those files because they can't be found, but Splunk should be now ready to start.

Repeat the operation for each index for which splunkd.log reports this type of error.

hexx
Splunk Employee
Splunk Employee

Glad to hear you're back online. If my answer was helpful, please validate it and give it an up vote.

0 Karma

MrSplunksta
Path Finder

Thanx a zillion for your help!
It really help me get it back working!
:-)

0 Karma

MrSplunksta
Path Finder

NOW FIXED!!!

Hi all!

I finally got my issue fixed by issuing the following command before upgrading to 4.1.3

C:\Program Files\Splunk\bin>splunk cmd recover-metadata D:\SplunkDB\SplunkDB\os\ db\hot_v1_2

Hope this helps others!

NOTE: Since my database is in a non default location, you'll have to ajust the command to point to YOUR splunk database location.

MrSplunksta
Path Finder

thx alot nick 🙂

0 Karma

sideview
SplunkTrust
SplunkTrust

You should probably accept the other answer I think. And this isnt an answer but you can click 'edit' to your question and add some UPDATE text at the bottom...

0 Karma

MrSplunksta
Path Finder

Just upgraded to 4.1.3 to no avail 😞

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...