Why is Splunk 6.1.1 with Splunk DB Connect 1.1.4 app crashing?

[imcadams]

I have a dedicated Windows 7 machine setup as a forwarder running Splunk 6.1.1 with Splunk DB Connect 1.1.4. Splunk keeps crashing almost daily and I need help to figure out what is causing the crash.

Here is the Splunk crash log:

[build 207789] 2014-07-16 00:31:58

 Access violation, cannot read at address [0x000000000000000F]

 Exception address: [0x000000013FB64B11]

 Crashing thread: MainTailingThread

    MxCsr:  [0x0000000000001FA0]

    SegDs:  [0x000000000000002B]

    SegEs:  [0x000000000000002B]

    SegFs:  [0x0000000000000053]

    SegGs:  [0x000000000000002B]

    SegSs:  [0x000000000000002B]

    SegCs:  [0x0000000000000033]

    EFlags:  [0x0000000000010206]

    Rsp:  [0x000000000C03D2D0]

    Rip:  [0x000000013FB64B11] ?

    Dr0:  [0x0000000000000000]

    Dr1:  [0x0000000000000000]

    Dr2:  [0x0000000000000000]

    Dr3:  [0x0000000000000000]

    Dr6:  [0x0000000000000000]

    Dr7:  [0x0000000000000000]

    Rax:  [0x000000000000000F]

    Rcx:  [0x00000000188A9938]

    Rdx:  [0x000000000C03E468]

    Rbx:  [0x000000000C03E3C0]

    Rbp:  [0x0000000000000000]

    Rsi:  [0x000000000C03E468]

    Rdi:  [0x0000000000000000]

    R8:  [0x000000000C03E3C0]

    R9:  [0x0000000000000000]

    R10:  [0x000000000E53C9F0]

    R11:  [0x000000000E53CAF0]

    R12:  [0x000000000C03E520]

    R13:  [0x0000000000000100]

    R14:  [0x00000000188A9938]

    R15:  [0x0000000000000000]

    DebugControl:  [0x0000000140BD3FB0]

    LastBranchToRip:  [0x0000000000000000]

    LastBranchFromRip:  [0x0000000000000000]

    LastExceptionToRip:  [0x0000000000000000]

    LastExceptionFromRip:  [0x0000000000000000]

 OS: Windows

 Arch: x86-64

 Backtrace:

  [0x000000013FB64B11] ?

Args:  [0x0000000000000000]  [0x0000000140BD3FB0]  [0x0000000000000100]

  [0x000000013F9311A5] ?

Args:  [0x0000000000000100]  [0x0000000000000196]  [0x000000000F733C88]

  [0x000000013F931616] ?

Args:  [0x000000000F7337D0]  [0x000000000F7337E2]  [0x000000000F7338D0]

  [0x000000013F931BAD] ?

Args:  [0x000000000F7336F0]  [0x000000000F733BD0]  [0x000000000F733BD0]

  [0x000000013F44C5ED] ?

Args:  [0xFFFFFFFFFFFFFFFE]  [0x00000000188A9930]  [0x000000000C03E639]

  [0x000000013F44A7E5] ?

Args:  [0x0000000006660B68]  [0x000000000F7336F0]  [0x0000000000000000]

  [0x000000013F44B875] ?

Args:  [0x0000000000000000]  [0x00000001404C7318]  [0x00000000004AB889]

  [0x000000013F44D61F] ?

Args:  [0x0000000006646840]  [0x00000000066467C8]  [0x0000000006646800]

  [0x000000013F442E41] ?

Args:  [0x0000000140BD3FB0]  [0x00000000066467C8]  [0x00000000066467C8]

  [0x000000013F443189] ?

Args:  [0x0000000006646840]  [0x00000001404C4A28]  [0x000000000C03EFF8]

  [0x000000013F43DF1B] ?

Args:  [0x01CFA0AEE1AD5E5C]  [0x01CFA0AEE1CBE2DC]  [0x000000000C03EFF8]

  [0x000000013FB77FD5] ?

Args:  [0x00000000000003E4]  [0x000000000C03EE10]  [0x01CFA0AEE1AD5E5C]

  [0x000000013FB2DAC5] ?

Args:  [0x0000000000000000]  [0x01CFA0AEE1C74EFC]  [0x0000000000000000]

  [0x000000013FB310AF] ?

Args:  [0x0000000000000000]  [0x01CFA0AEE1AD5E5C]  [0x000000000C03FA20]

  [0x000000013F44374B] ?

Args:  [0x0000000004A280D0]  [0x0000000006630E08]  [0x0000000004A280D0]

  [0x000000013F441118] ?

Args:  [0x0000000004A280D0]  [0x0000000004A280D0]  [0x0000000004130470]

  [0x000000013FAFE02C] ?

Args:  [0x0000000004A280D0]  [0x000007FEF83D432B]  [0x0000000000000000]

  [0x000000013F3E3A57] ?

Args:  [0x0000000004130470]  [0x0000000000000000]  [0x0000000000000000]

  [0x000007FEF83D3FEF] beginthreadex + 263/284

Args:  [0x000007FEF8471DB0]  [0x0000000000000000]  [0x0000000000000000]

  [0x000007FEF83D4196] endthreadex + 402/404

Args:  [0x0000000000000000]  [0x0000000000000000]  [0x0000000000000000]

  [0x0000000076C7652D] BaseThreadInitThunk + 13/96

Args:  [0x0000000000000000]  [0x0000000000000000]  [0x0000000000000000]

  [0x0000000076EAC541] RtlUserThreadStart + 33/1024

Args:  [0x0000000000000000]  [0x0000000000000000]  [0x0000000000000000]

 Crash dump written to: C:\Program Files\Splunk\var\log\splunk\C__Program Files_Splunk_bin_splunkd_exe_crash-2014-07-16-00-31-58.dmp

xxSVR /6.1 Service Pack 1

GetLastError(): 0

Threads running: 54

argv: [Splunkd -p 8089]

Thread: "MainTailingThread", did_join=0, ready_to_run=Y, main_thread=N

First 4 bytes of Thread token @0000000004A280E4:

00000000  b4 14 00 00                                       |....|

00000004

First 8 bytes of Timeout object @000000000C03EFF8:

00000000  38 15 5e 40 01 00 00 00                           |8.^@....|

00000008

FilesystemChangeWatcher: _timeoutActive=Y, _throttled=N, _waitingForNotifyCount=1

  EMPTY Q: waitingForTimeout=N, noAction=N, stat=Y, immediateStat=Y, readdir=Y, notify=Y

WatchedTailFile-WatchedFileState: path="C:\Program Files\Splunk\var\log\introspection\resource_usage.log", flags=0xAB

First 36 bytes of PathnameStat @0000000006646860:

00000000  20 00 00 00 d4 11 59 ec  3c 95 cf 01 ce 30 6a cb  | .....Y.<....0j.|

00000010  8b a0 cf 01 86 dc 1d 65  ae a0 cf 01 00 00 00 00  |.......e........|

00000020  89 b8 4a 00                                       |..J.|

00000024

FilesystemChangeWatcher: _timeoutActive=Y, _throttled=N, _waitingForNotifyCount=1

  EMPTY Q: waitingForTimeout=N, noAction=N, stat=Y, immediateStat=Y, readdir=Y, notify=Y

  Timeout: _when = 01CFA0AEE1AFFE2B, _initialMsec = 1000

file-in: _initialized=Y, _lastCharWasNewline=Y, _lastReadHadNulls=N, _wasCrcConflict=N, _warned=N

         _nullsWarned=N, _wasTooNew=N, _exists=Y, _noDebug=N

         _hadExplicitSource=N, _crossedInitCrcLenBoundary=N, _classifiedAtLeastOnce=Y, _fileReplaced=N, _readPathAfterRealEOF=N

         _onlyNotifiedOnce=N, _isArchive=N, _isCached=111213, _unowned=N, _deleteOnEOF=N

         _overrideDeleteOnEOF=N, _doNotDeleteChildren=N, _alwaysReopen=N, _readFromEnd=N, _readIrregardless=N

         _fileCheckMethod=0, _crcSalt=, _origPath=

         _bytesRead=4893640, _storingBytesRead=0, _initCrc=0xe44136de9235d455, _seekCrc=0x962ee9e3e38b475b

         _filenameCrc=0xea96444b68dd433d, _fallbackCrc=0x0, _lastEOFTime=, _modTime=01CFA0AE651DDC86

         _eofSeconds=3, _ignoreThresh=, _initCrcBytes=256, _initCrcForBatch=0x0

         _pendingMetadata=[st: ["splunk_resource_usage"]; csets:["UTF-8"]]

         _prevFd=18446744073709551615{invalid}, _pdModels=[1 PD: [PD: flags=0xAA0030, [_path] = "C:\Program Files\Splunk\var\log\introspection\resource_usage.log", [_MetaData:Index] = "_introspection", [evt_resolve_ad_obj] = "0", [MetaData:Source] = "source::C:\Program Files\Splunk\var\log\introspection\resource_usage.log", [MetaData:Host] = "host::xxSVR", [MetaData:Sourcetype] = "sourcetype::splunk_resource_usage", [_hpn] = "_hpn", [_charSet] = "UTF-8", [_conf] = "source::C:\Program Files\Splunk\var\log\introspection\resource_usage.log|host::xxSVR|splunk_resource_usage|406297", [_channel] = "406297"]]

         _rescheduleDelay=1000, _rescheduleTarget=, _name=C:\Program Files\Splunk\var\log\introspection\resource_usage.log, _statusName=

         _st=[REG: size=4896905, mtime=01CFA0AE651DDC86]

         _toStringPrefix=state=0x00000000066467C8, _backoff=0

         _stdataInputHeaderProcessing=[]

         _tmpExtractionsConf=mode=8 HEADER_FIELD_LINE_NUMBER=0 HEADER_FIELD_DELIMITER=',' HEADER_FIELD_QUOTE='"' FIELD_DELIMITER=',' FIELD_QUOTE='"'

         _detectTrailingNulls=Y, _detectReadingFromOffSet=Y, _readAndSkipHeader=N, _uniqueId=406297

         _sourceStanza=source::C:\Program Files\Splunk\var\log\introspection\resource_usage.log

         _sourceProps={ANNOTATE_PUNCT -> True, BREAK_ONLY_BEFORE -> , BREAK_ONLY_BEFORE_DATE -> True, CHARSET -> AUTO, DATETIME_CONFIG -> \etc\datetime.xml, HEADER_MODE -> , LEARN_SOURCETYPE -> true, LINE_BREAKER_LOOKBEHIND -> 100, MAX_DAYS_AGO -> 2000, MAX_DAYS_HENCE -> 2, MAX_DIFF_SECS_AGO -> 3600, MAX_DIFF_SECS_HENCE -> 604800, MAX_EVENTS -> 256, MAX_TIMESTAMP_LOOKAHEAD -> 128, MUST_BREAK_AFTER -> , MUST_NOT_BREAK_AFTER -> , MUST_NOT_BREAK_BEFORE -> , SEGMENTATION -> indexing, SEGMENTATION-all -> full, SEGMENTATION-inner -> inner, SEGMENTATION-outer -> outer, SEGMENTATION-raw -> none, SEGMENTATION-standard -> standard, SHOULD_LINEMERGE -> True, TRANSFORMS -> , TRUNCATE -> 10000, detect_trailing_nulls -> auto, maxDist -> 100, sourcetype -> splunk_resource_usage, unarchive_cmd -> }

  _rawPath=$SPLUNK_HOME\var\log\introspection

x86 CPUID registers:

         0: 0000000D 756E6547 6C65746E 49656E69

         1: 000306A9 00100800 7FBAE3BF BFEBFBFF

         2: 76035A01 00F0B0FF 00000000 00CA0000

         3: 00000000 00000000 00000000 00000000

         4: 1C004121 01C0003F 0000003F 00000000

         5: 00000040 00000040 00000003 00001120

         6: 00000077 00000002 00000009 00000000

         7: 00000000 00000281 00000000 00000000

         8: 00000000 00000000 00000000 00000000

         9: 00000000 00000000 00000000 00000000

         A: 07300803 00000000 00000000 00000603

         B: 00000001 00000001 00000100 00000000

         C: 00000000 00000000 00000000 00000000

         😧 00000007 00000340 00000340 00000000

  80000000: 80000008 00000000 00000000 00000000

  80000001: 00000000 00000000 00000001 28100800

  80000002: 20202020 20202020 65746E49 2952286C

  80000003: 726F4320 4D542865 35692029 3333332D

  80000004: 50432030 20402055 30302E33 007A4847

  80000005: 00000000 00000000 00000000 00000000

  80000006: 00000000 00000000 01006040 00000000

  80000007: 00000000 00000000 00000000 00000100

  80000008: 00003024 00000000 00000000 00000000

terminating...

Here is the windows error log:

LogName=Application

SourceName=Application Error

EventCode=1000

EventType=2

Type=Error

ComputerName=xxSVR

TaskCategory=Application Crashing Events

OpCode=Info

RecordNumber=9122

Keywords=Classic

Message=Faulting application name: splunkd.exe, version: 1537.256.0.11181, time stamp: 0x536c0b96

Faulting module name: splunkd.exe, version: 1537.256.0.11181, time stamp: 0x536c0b96

Exception code: 0xc0000005

Fault offset: 0x00000000007b4b11

Faulting process id: 0x21ac

Faulting application start time: 0x01cf9fa17c867f47

Faulting application path: C:\Program Files\Splunk\bin\splunkd.exe

Faulting module path: C:\Program Files\Splunk\bin\splunkd.exe

Report Id: 23cbbaff-0ca2-11e4-97ee-a41f726eb696

Here are the log events from the splunkd.log surrounding the crash time:

07-16-2014 00:30:36.059 -0400 INFO  WatchedFile - Will begin reading at offset=10447766 for file='C:\Program Files\Splunk\var\log\splunk\dbx.log.1'.

07-16-2014 00:30:37.510 -0400 INFO  WatchedFile - Will begin reading at offset=0 for file='C:\Program Files\Splunk\var\log\splunk\dbx.log'.

07-16-2014 07:06:14.994 -0400 INFO  loader - win-service: Starting as a Windows service: will run various system checks first...

07-16-2014 07:06:14.994 -0400 INFO  loader - Automatic migration of modular inputs

[davebrooking · Accepted solution]

I think you've come up against a known problem with the 'new' introspection functionality. Take a look at this Answer
http://answers.splunk.com/answers/137438/i-upgraded-to-61-and-now-splunk-is-crashing-while-reading-m...

I believe it's due to be fixed in 6.1.3, so you'll need to apply the workaround until that's available.

Dave

[imcadams]

Yes I did. I will update above with the log events near the time the application crashed.

[changwoo]

Did you check the splunkd.log ?