Splunkd won't restart

[MrSplunksta]

Hi all!

This windows splunk ver splunk-4.1.2-79191-x86-release running on a windows server 2003 US does not start anymore.

Does anyone know what I should do to fix this problem?

Thanks in advance for your time 🙂

Here's what I know: - the server rebooted by itself following a nessus 4 scan (ouch) - the splunkd service now hangs at each start - splunkd.log reports the following: 06-08-2010 13:29:35.062 ERROR WordPositionData - couldn't parse hash code: - there is a crash log file that contains the following.

[build 79191] C++ exception: object@[0x01CBEB64], type@[0x00D0F58C] Exception is Non-continuable Exception address: [0x77E4BEF7] Crashing thread: indexerPipe ContextFlags: [0x00010007] Dr0: [0x78180475] Dr1: [0x015652AB] Dr2: [0x013C03C1] Dr3: [0x00000001] Dr6: [0x00000014] Dr7: [0x01CBE7F8] SegGs: [0x01CB0000] SegFs: [0x0000003B] SegEs: [0x01CB0023] SegDs: [0x7C820023] Edi: [0x01CBECC8] Esi: [0x01CBEB38] Ebx: [0xFFFFFFFF] Edx: [0x01CBEB64] Ecx: [0x00000000] Eax: [0x01CBEAB0] Ebp: [0x01CBEB00] Eip: [0x77E4BEF7] RaiseException + 60/87 SegCs: [0x011A001B] EFlags: [0x00000206] Esp: [0x01CBEAAC] SegSs: [0x78180023]

OS: Windows Arch: i386

Backtrace: Frame 0 @[0x01CBEB00]: [0x78158E89] CxxThrowException + 70/77 Frame 1 @[0x01CBEB38]: [0x006C64AE] ? Frame 2 @[0x01CBEDFC]: [0x01CBF3A8] ? Frame 3 @[0x00B70610]: (Frame below stack)

Crash dump written to: C:\Program Files\Splunk\var\log\splunk\C__Program Files_Splunk_bin_splunkd_exe_crash-2010-06-08-13-32-30.dmp

ENLIL /5.2 Service Pack 2 C++ Exception type: WordPositionData::Exception -> std::exception what(): couldn't parse hash code: Threads running: 13 terminating...

[hexx]

The presence of "ERROR WordPositionData - couldn't parse hash code:" messages in splunkd.log often indicates an inconsistency in one of the metadata files (Hosts.data, Sources.data, SourceTypes.data) located in the hot/warm index repository (Example for the main index : $SPLUNK_DB/defaultdb/db/) or in one of the buckets (usually one of the hot ones) contained in that index.

To fix this, the first thing to do is to identify which metadata file(s) has/have inconsistencies.

To that effect, the following command has to be run for the incriminated index (check splunkd.log, it's the index that was just being opened before splunkd crashed) and for all of it's hot/warm buckets :

$SPLUNK_HOME/bin/recover-metadata {path_to_index|path_to_bucket} --validate

For a given index, I like to run the two commands below to check the metadata files at the root of the hot/warm db first, and then each bucket using the list from .bucketManifest :

$SPLUNK_HOME/bin/recover-metadata $SPLUNK_DB/{index_name}/db/ --validate

for i in 'cat $SPLUNK_DB/{index_name}/db/.bucketManifest | cut -f3 -d " "'; do $SPLUNK_HOME/bin/recover-metadata $SPLUNK_DB/{index_name}/db/$i ; done

Each time an error is reported, the corresponding .data file should be moved or deleted. Once all corrupted metadata files have been removed, the check should be run again. It will indicate errors for those files because they can't be found, but Splunk should be now ready to start.

Repeat the operation for each index for which splunkd.log reports this type of error.

[hexx]

Glad to hear you're back online. If my answer was helpful, please validate it and give it an up vote.

[MrSplunksta]

Thanx a zillion for your help!
It really help me get it back working!
:-)

[MrSplunksta · Accepted solution]

NOW FIXED!!!

Hi all!

I finally got my issue fixed by issuing the following command before upgrading to 4.1.3

C:\Program Files\Splunk\bin>splunk cmd recover-metadata D:\SplunkDB\SplunkDB\os\ db\hot_v1_2

Hope this helps others!

NOTE: Since my database is in a non default location, you'll have to ajust the command to point to YOUR splunk database location.

[MrSplunksta]

thx alot nick 🙂

[sideview]

You should probably accept the other answer I think. And this isnt an answer but you can click 'edit' to your question and add some UPDATE text at the bottom...

[MrSplunksta]

Just upgraded to 4.1.3 to no avail 😞