Monitoring Splunk

Splunkd crash after upgrade to 5.0.1

rush05
Engager

Hi, we just upgraded splunk from 4.3.1 to 5.0.1 and now every time we start splunk, splunkd crashes.

Here is the text from the crashlog:

[build 143156] 2013-01-21 11:29:10
Received fatal signal 6 (Abort).
 Cause:
   Unknown signal origin (si_code=-1).
 Crashing thread: MainTailingThread
 Registers:
    RIP:  [0xFFFFFD7FFEAE314A] __lwp_kill + 10 (/lib/amd64/libc.so.1)
    RDI:  [0x0000000000000032]
    RSI:  [0x0000000000000006]
    RBP:  [0xFFFFFD7FF53FF350]
    RSP:  [0xFFFFFD7FF53FF348]
    RAX:  [0x0000000000000000]
    RBX:  [0x0000000000000006]
    RCX:  [0x0000000000000005]
    RDX:  [0xFFFFFEBFB492CBE0]
    R8:  [0x0000000000000001]
    R9:  [0x0000000000000000]
    R10:  [0x0000000000000005]
    R11:  [0x0000000000000020]
    R12:  [0x00000000000000E5]
    R13:  [0x0000000001CC4E98]
    R14:  [0x0000000001AEA230]
    R15:  [0xFFFFFD7FF53FF840]
    RFL:  [0x0000000000000286]
    TRAPNO:  [0x000000000000000E]
    ERR:  [0x0000000000000004]
    CS:  [0x000000000000004B]
    GS:  [0x0000000000000000]
    FS:  [0x0000000000000000]

 OS: SunOS
 Arch: x86-64

 Backtrace:
  [0xFFFFFFFFFFFFFFFF] ?
  [0xFFFFFD7FFEA87F19] raise + 25 (/lib/amd64/libc.so.1)
  [0xFFFFFD7FFEA669CE] abort + 94 (/lib/amd64/libc.so.1)
  [0xFFFFFD7FFEA66C8E] _assert + 126 (/lib/amd64/libc.so.1)
  [0x00000000009D1148] _ZN16FileInputTracker10computeCRCEPm14FileDescriptorRK3Strll + 920 (/opt/splunk/bin/splunkd)
  [0x00000000009D1656] _ZN16FileInputTracker11fileHalfMd5EPm14FileDescriptorRK3Strll + 22 (/opt/splunk/bin/splunkd)
  [0x00000000009EA99A] _ZN3WTF13loadFishStateEb + 650 (/opt/splunk/bin/splunkd)
  [0x00000000009DDAB4] _ZN10TailReader8readFileER15WatchedTailFileP11TailWatcher + 148 (/opt/splunk/bin/splunkd)
  [0x00000000009DDCDE] _ZN11TailWatcher8readFileER15WatchedTailFile + 254 (/opt/splunk/bin/splunkd)
  [0x00000000009DFD92] _ZN11TailWatcher11fileChangedEP16WatchedFileStateRK7Timeval + 482 (/opt/splunk/bin/splunkd)
  [0x0000000000EBFC84] _ZN30FilesystemChangeInternalWorker15callFileChangedER7TimevalP16WatchedFileState + 148 (/opt/splunk/bin/splunkd)
  [0x0000000000EC1743] _ZN30FilesystemChangeInternalWorker12when_expiredERy + 499 (/opt/splunk/bin/splunkd)
  [0x0000000000F18634] _ZN11TimeoutHeap18runExpiredTimeoutsER7Timeval + 180 (/opt/splunk/bin/splunkd)
  [0x0000000000EBABBC] _ZN9EventLoop3runEv + 188 (/opt/splunk/bin/splunkd)
  [0x00000000009E6B70] _ZN11TailWatcher3runEv + 144 (/opt/splunk/bin/splunkd)
  [0x00000000009E6CD6] _ZN13TailingThread4mainEv + 278 (/opt/splunk/bin/splunkd)
  [0x0000000000F16012] _ZN6Thread8callMainEPv + 98 (/opt/splunk/bin/splunkd)
  [0xFFFFFD7FFEADD60B] _thr_slot_offset + 795 (/lib/amd64/libc.so.1)
  [0xFFFFFD7FFEADD840] smt_pause + 96 (/lib/amd64/libc.so.1)
 SunOS / dtslabmgt04 / 5.10 / Generic_147441-24 / i86pc
 Last few lines of stderr (may contain info on assertion failure, but also could be old):
    2013-01-18 10:10:25.499 -0500 Interrupt signal received
    2013-01-18 11:53:06.183 -0500 splunkd started (build 115073)
    2013-01-21 10:23:51.077 -0500 Interrupt signal received
    2013-01-21 11:26:26.127 -0500 splunkd started (build 143156)
    Assertion failed: bytesToHash < 1048576, file /opt/splunk/p4/splunk/branches/5.0.1/src/pipeline/input/FileInputTracker.cpp, line 229
    2013-01-21 11:27:50.445 -0500 splunkd started (build 143156)
    Assertion failed: bytesToHash < 1048576, file /opt/splunk/p4/splunk/branches/5.0.1/src/pipeline/input/FileInputTracker.cpp, line 229
    2013-01-21 11:29:08.706 -0500 splunkd started (build 143156)
    Assertion failed: bytesToHash < 1048576, file /opt/splunk/p4/splunk/branches/5.0.1/src/pipeline/input/FileInputTracker.cpp, line 229

Threads running: 40
argv: [splunkd -p 8089 start]
terminating...

If anyone can help, it would be much appreciated. This is on our test server, but we are supposed to roll it out into production soon.

Thanks!

1 Solution

jbsplunk
Splunk Employee
Splunk Employee

This is a known issue, SPL-58292, which has been reported to support and is presently being investigated. At this time, there aren't any workarounds.

View solution in original post

jbsplunk
Splunk Employee
Splunk Employee

This is a known issue, SPL-58292, which has been reported to support and is presently being investigated. At this time, there aren't any workarounds.

jbsplunk
Splunk Employee
Splunk Employee

You can file a case with support if you have an enterprise contract and they'll be able to provide you with additional details.

0 Karma

rush05
Engager

Thank you for your reply. Where would I go to find more details on SPL-58292?

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to May Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars this month. This ...

They're back! Join the SplunkTrust and MVP at .conf24

With our highly anticipated annual conference, .conf, comes the fez-wearers you can trust! The SplunkTrust, as ...

Enterprise Security Content Update (ESCU) | New Releases

Last month, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...