I have a dedicated Windows 7 machine setup as a forwarder running Splunk 6.1.1 with Splunk DB Connect 1.1.4. Splunk keeps crashing almost daily and I need help to figure out what is causing the crash.
Here is the Splunk crash log:
[build 207789] 2014-07-16 00:31:58
Access violation, cannot read at address [0x000000000000000F]
Exception address: [0x000000013FB64B11]
Crashing thread: MainTailingThread
MxCsr: [0x0000000000001FA0]
SegDs: [0x000000000000002B]
SegEs: [0x000000000000002B]
SegFs: [0x0000000000000053]
SegGs: [0x000000000000002B]
SegSs: [0x000000000000002B]
SegCs: [0x0000000000000033]
EFlags: [0x0000000000010206]
Rsp: [0x000000000C03D2D0]
Rip: [0x000000013FB64B11] ?
Dr0: [0x0000000000000000]
Dr1: [0x0000000000000000]
Dr2: [0x0000000000000000]
Dr3: [0x0000000000000000]
Dr6: [0x0000000000000000]
Dr7: [0x0000000000000000]
Rax: [0x000000000000000F]
Rcx: [0x00000000188A9938]
Rdx: [0x000000000C03E468]
Rbx: [0x000000000C03E3C0]
Rbp: [0x0000000000000000]
Rsi: [0x000000000C03E468]
Rdi: [0x0000000000000000]
R8: [0x000000000C03E3C0]
R9: [0x0000000000000000]
R10: [0x000000000E53C9F0]
R11: [0x000000000E53CAF0]
R12: [0x000000000C03E520]
R13: [0x0000000000000100]
R14: [0x00000000188A9938]
R15: [0x0000000000000000]
DebugControl: [0x0000000140BD3FB0]
LastBranchToRip: [0x0000000000000000]
LastBranchFromRip: [0x0000000000000000]
LastExceptionToRip: [0x0000000000000000]
LastExceptionFromRip: [0x0000000000000000]
OS: Windows
Arch: x86-64
Backtrace:
[0x000000013FB64B11] ?
Args: [0x0000000000000000] [0x0000000140BD3FB0] [0x0000000000000100]
[0x000000013F9311A5] ?
Args: [0x0000000000000100] [0x0000000000000196] [0x000000000F733C88]
[0x000000013F931616] ?
Args: [0x000000000F7337D0] [0x000000000F7337E2] [0x000000000F7338D0]
[0x000000013F931BAD] ?
Args: [0x000000000F7336F0] [0x000000000F733BD0] [0x000000000F733BD0]
[0x000000013F44C5ED] ?
Args: [0xFFFFFFFFFFFFFFFE] [0x00000000188A9930] [0x000000000C03E639]
[0x000000013F44A7E5] ?
Args: [0x0000000006660B68] [0x000000000F7336F0] [0x0000000000000000]
[0x000000013F44B875] ?
Args: [0x0000000000000000] [0x00000001404C7318] [0x00000000004AB889]
[0x000000013F44D61F] ?
Args: [0x0000000006646840] [0x00000000066467C8] [0x0000000006646800]
[0x000000013F442E41] ?
Args: [0x0000000140BD3FB0] [0x00000000066467C8] [0x00000000066467C8]
[0x000000013F443189] ?
Args: [0x0000000006646840] [0x00000001404C4A28] [0x000000000C03EFF8]
[0x000000013F43DF1B] ?
Args: [0x01CFA0AEE1AD5E5C] [0x01CFA0AEE1CBE2DC] [0x000000000C03EFF8]
[0x000000013FB77FD5] ?
Args: [0x00000000000003E4] [0x000000000C03EE10] [0x01CFA0AEE1AD5E5C]
[0x000000013FB2DAC5] ?
Args: [0x0000000000000000] [0x01CFA0AEE1C74EFC] [0x0000000000000000]
[0x000000013FB310AF] ?
Args: [0x0000000000000000] [0x01CFA0AEE1AD5E5C] [0x000000000C03FA20]
[0x000000013F44374B] ?
Args: [0x0000000004A280D0] [0x0000000006630E08] [0x0000000004A280D0]
[0x000000013F441118] ?
Args: [0x0000000004A280D0] [0x0000000004A280D0] [0x0000000004130470]
[0x000000013FAFE02C] ?
Args: [0x0000000004A280D0] [0x000007FEF83D432B] [0x0000000000000000]
[0x000000013F3E3A57] ?
Args: [0x0000000004130470] [0x0000000000000000] [0x0000000000000000]
[0x000007FEF83D3FEF] beginthreadex + 263/284
Args: [0x000007FEF8471DB0] [0x0000000000000000] [0x0000000000000000]
[0x000007FEF83D4196] endthreadex + 402/404
Args: [0x0000000000000000] [0x0000000000000000] [0x0000000000000000]
[0x0000000076C7652D] BaseThreadInitThunk + 13/96
Args: [0x0000000000000000] [0x0000000000000000] [0x0000000000000000]
[0x0000000076EAC541] RtlUserThreadStart + 33/1024
Args: [0x0000000000000000] [0x0000000000000000] [0x0000000000000000]
Crash dump written to: C:\Program Files\Splunk\var\log\splunk\C__Program Files_Splunk_bin_splunkd_exe_crash-2014-07-16-00-31-58.dmp
xxSVR /6.1 Service Pack 1
GetLastError(): 0
Threads running: 54
argv: [Splunkd -p 8089]
Thread: "MainTailingThread", did_join=0, ready_to_run=Y, main_thread=N
First 4 bytes of Thread token @0000000004A280E4:
00000000 b4 14 00 00 |....|
00000004
First 8 bytes of Timeout object @000000000C03EFF8:
00000000 38 15 5e 40 01 00 00 00 |8.^@....|
00000008
FilesystemChangeWatcher: _timeoutActive=Y, _throttled=N, _waitingForNotifyCount=1
EMPTY Q: waitingForTimeout=N, noAction=N, stat=Y, immediateStat=Y, readdir=Y, notify=Y
WatchedTailFile-WatchedFileState: path="C:\Program Files\Splunk\var\log\introspection\resource_usage.log", flags=0xAB
First 36 bytes of PathnameStat @0000000006646860:
00000000 20 00 00 00 d4 11 59 ec 3c 95 cf 01 ce 30 6a cb | .....Y.<....0j.|
00000010 8b a0 cf 01 86 dc 1d 65 ae a0 cf 01 00 00 00 00 |.......e........|
00000020 89 b8 4a 00 |..J.|
00000024
FilesystemChangeWatcher: _timeoutActive=Y, _throttled=N, _waitingForNotifyCount=1
EMPTY Q: waitingForTimeout=N, noAction=N, stat=Y, immediateStat=Y, readdir=Y, notify=Y
Timeout: _when = 01CFA0AEE1AFFE2B, _initialMsec = 1000
file-in: _initialized=Y, _lastCharWasNewline=Y, _lastReadHadNulls=N, _wasCrcConflict=N, _warned=N
_nullsWarned=N, _wasTooNew=N, _exists=Y, _noDebug=N
_hadExplicitSource=N, _crossedInitCrcLenBoundary=N, _classifiedAtLeastOnce=Y, _fileReplaced=N, _readPathAfterRealEOF=N
_onlyNotifiedOnce=N, _isArchive=N, _isCached=111213, _unowned=N, _deleteOnEOF=N
_overrideDeleteOnEOF=N, _doNotDeleteChildren=N, _alwaysReopen=N, _readFromEnd=N, _readIrregardless=N
_fileCheckMethod=0, _crcSalt=, _origPath=
_bytesRead=4893640, _storingBytesRead=0, _initCrc=0xe44136de9235d455, _seekCrc=0x962ee9e3e38b475b
_filenameCrc=0xea96444b68dd433d, _fallbackCrc=0x0, _lastEOFTime=, _modTime=01CFA0AE651DDC86
_eofSeconds=3, _ignoreThresh=, _initCrcBytes=256, _initCrcForBatch=0x0
_pendingMetadata=[st: ["splunk_resource_usage"]; csets:["UTF-8"]]
_prevFd=18446744073709551615{invalid}, _pdModels=[1 PD: [PD: flags=0xAA0030, [_path] = "C:\Program Files\Splunk\var\log\introspection\resource_usage.log", [_MetaData:Index] = "_introspection", [evt_resolve_ad_obj] = "0", [MetaData:Source] = "source::C:\Program Files\Splunk\var\log\introspection\resource_usage.log", [MetaData:Host] = "host::xxSVR", [MetaData:Sourcetype] = "sourcetype::splunk_resource_usage", [_hpn] = "_hpn", [_charSet] = "UTF-8", [_conf] = "source::C:\Program Files\Splunk\var\log\introspection\resource_usage.log|host::xxSVR|splunk_resource_usage|406297", [_channel] = "406297"]]
_rescheduleDelay=1000, _rescheduleTarget=, _name=C:\Program Files\Splunk\var\log\introspection\resource_usage.log, _statusName=
_st=[REG: size=4896905, mtime=01CFA0AE651DDC86]
_toStringPrefix=state=0x00000000066467C8, _backoff=0
_stdataInputHeaderProcessing=[]
_tmpExtractionsConf=mode=8 HEADER_FIELD_LINE_NUMBER=0 HEADER_FIELD_DELIMITER=',' HEADER_FIELD_QUOTE='"' FIELD_DELIMITER=',' FIELD_QUOTE='"'
_detectTrailingNulls=Y, _detectReadingFromOffSet=Y, _readAndSkipHeader=N, _uniqueId=406297
_sourceStanza=source::C:\Program Files\Splunk\var\log\introspection\resource_usage.log
_sourceProps={ANNOTATE_PUNCT -> True, BREAK_ONLY_BEFORE -> , BREAK_ONLY_BEFORE_DATE -> True, CHARSET -> AUTO, DATETIME_CONFIG -> \etc\datetime.xml, HEADER_MODE -> , LEARN_SOURCETYPE -> true, LINE_BREAKER_LOOKBEHIND -> 100, MAX_DAYS_AGO -> 2000, MAX_DAYS_HENCE -> 2, MAX_DIFF_SECS_AGO -> 3600, MAX_DIFF_SECS_HENCE -> 604800, MAX_EVENTS -> 256, MAX_TIMESTAMP_LOOKAHEAD -> 128, MUST_BREAK_AFTER -> , MUST_NOT_BREAK_AFTER -> , MUST_NOT_BREAK_BEFORE -> , SEGMENTATION -> indexing, SEGMENTATION-all -> full, SEGMENTATION-inner -> inner, SEGMENTATION-outer -> outer, SEGMENTATION-raw -> none, SEGMENTATION-standard -> standard, SHOULD_LINEMERGE -> True, TRANSFORMS -> , TRUNCATE -> 10000, detect_trailing_nulls -> auto, maxDist -> 100, sourcetype -> splunk_resource_usage, unarchive_cmd -> }
_rawPath=$SPLUNK_HOME\var\log\introspection
x86 CPUID registers:
0: 0000000D 756E6547 6C65746E 49656E69
1: 000306A9 00100800 7FBAE3BF BFEBFBFF
2: 76035A01 00F0B0FF 00000000 00CA0000
3: 00000000 00000000 00000000 00000000
4: 1C004121 01C0003F 0000003F 00000000
5: 00000040 00000040 00000003 00001120
6: 00000077 00000002 00000009 00000000
7: 00000000 00000281 00000000 00000000
8: 00000000 00000000 00000000 00000000
9: 00000000 00000000 00000000 00000000
A: 07300803 00000000 00000000 00000603
B: 00000001 00000001 00000100 00000000
C: 00000000 00000000 00000000 00000000
😧 00000007 00000340 00000340 00000000
80000000: 80000008 00000000 00000000 00000000
80000001: 00000000 00000000 00000001 28100800
80000002: 20202020 20202020 65746E49 2952286C
80000003: 726F4320 4D542865 35692029 3333332D
80000004: 50432030 20402055 30302E33 007A4847
80000005: 00000000 00000000 00000000 00000000
80000006: 00000000 00000000 01006040 00000000
80000007: 00000000 00000000 00000000 00000100
80000008: 00003024 00000000 00000000 00000000
terminating...
LogName=Application
SourceName=Application Error
EventCode=1000
EventType=2
Type=Error
ComputerName=xxSVR
TaskCategory=Application Crashing Events
OpCode=Info
RecordNumber=9122
Keywords=Classic
Message=Faulting application name: splunkd.exe, version: 1537.256.0.11181, time stamp: 0x536c0b96
Faulting module name: splunkd.exe, version: 1537.256.0.11181, time stamp: 0x536c0b96
Exception code: 0xc0000005
Fault offset: 0x00000000007b4b11
Faulting process id: 0x21ac
Faulting application start time: 0x01cf9fa17c867f47
Faulting application path: C:\Program Files\Splunk\bin\splunkd.exe
Faulting module path: C:\Program Files\Splunk\bin\splunkd.exe
Report Id: 23cbbaff-0ca2-11e4-97ee-a41f726eb696
Here are the log events from the splunkd.log surrounding the crash time:
07-16-2014 00:30:36.059 -0400 INFO WatchedFile - Will begin reading at offset=10447766 for file='C:\Program Files\Splunk\var\log\splunk\dbx.log.1'.
07-16-2014 00:30:37.510 -0400 INFO WatchedFile - Will begin reading at offset=0 for file='C:\Program Files\Splunk\var\log\splunk\dbx.log'.
07-16-2014 07:06:14.994 -0400 INFO loader - win-service: Starting as a Windows service: will run various system checks first...
07-16-2014 07:06:14.994 -0400 INFO loader - Automatic migration of modular inputs
I think you've come up against a known problem with the 'new' introspection functionality. Take a look at this Answer
http://answers.splunk.com/answers/137438/i-upgraded-to-61-and-now-splunk-is-crashing-while-reading-m...
I believe it's due to be fixed in 6.1.3, so you'll need to apply the workaround until that's available.
Dave
I think you've come up against a known problem with the 'new' introspection functionality. Take a look at this Answer
http://answers.splunk.com/answers/137438/i-upgraded-to-61-and-now-splunk-is-crashing-while-reading-m...
I believe it's due to be fixed in 6.1.3, so you'll need to apply the workaround until that's available.
Dave
Yes I did. I will update above with the log events near the time the application crashed.
Did you check the splunkd.log ?