Splunk failed to start after upgrading from V4.3.2 to V5.0.2

[tsunamii]


[build 149561] 2013-02-15 08:52:24
Received fatal signal 6 (Aborted).
Cause:
Signal sent by PID 15468 running under UID 579.
Crashing thread: MainTailingThread
Registers:
RIP: [0x00007F3A3FB7D8A5] gsignal + 53 (/lib64/libc.so.6)
RDI: [0x0000000000003C6C]
RSI: [0x0000000000003C9B]
RBP: [0x0000000001307210]
RSP: [0x00007F3A35FF6278]
RAX: [0x0000000000000000]
RBX: [0x00007F3A417A5000]
RCX: [0xFFFFFFFFFFFFFFFF]
RDX: [0x0000000000000006]
R8: [0x00007F3A3F400000]
R9: [0x00007F3A4029AEB0]
R10: [0x0000000000000008]
R11: [0x0000000000000206]
R12: [0x0000000001307930]
R13: [0x0000000001306DE0]
R14: [0x00007F3A38438288]
R15: [0x00007F3A384381C8]
EFL: [0x0000000000000206]
TRAPNO: [0x0000000000000000]
ERR: [0x0000000000000000]
CSGSFS: [0x0000000000000033]
OLDMASK: [0x0000000000000000]

OS: Linux
Arch: x86-64

Backtrace:
[0x00007F3A3FB7D8A5] gsignal + 53 (/lib64/libc.so.6)
[0x00007F3A3FB7F085] abort + 373 (/lib64/libc.so.6)
[0x00007F3A3FB76A1E] ? (/lib64/libc.so.6)
[0x00007F3A3FB76AE0] __assert_perror_fail + 0 (/lib64/libc.so.6)
[0x00000000006FCD42] _ZN16FileInputTracker10computeCRCEPm14FileDescriptorRK3Strll + 1906 (splunkd)
[0x00000000006FCE71] _ZN16FileInputTracker11fileHalfMd5EPm14FileDescriptorRK3Strll + 17 (splunkd)
[0x000000000071B844] _ZN3WTF13loadFishStateEb + 644 (splunkd)
[0x000000000070A6C5] _ZN10TailReader8readFileER15WatchedTailFileP11TailWatcher + 149 (splunkd)
[0x000000000070A8E4] _ZN11TailWatcher8readFileER15WatchedTailFile + 260 (splunkd)
[0x000000000070C9FB] _ZN11TailWatcher11fileChangedEP16WatchedFileStateRK7Timeval + 363 (splunkd)
[0x0000000000D3F4E1] _ZN30FilesystemChangeInternalWorker15callFileChangedER7TimevalP16WatchedFileState + 113 (splunkd)
[0x0000000000D40DCF] _ZN30FilesystemChangeInternalWorker12when_expiredERy + 479 (splunkd)
[0x0000000000DA5553] _ZN11TimeoutHeap18runExpiredTimeoutsER7Timeval + 227 (splunkd)
[0x0000000000D3A318] _ZN9EventLoop3runEv + 216 (splunkd)
[0x000000000071328F] _ZN11TailWatcher3runEv + 143 (splunkd)
[0x00000000007133EB] _ZN13TailingThread4mainEv + 267 (splunkd)
[0x0000000000DA2F32] _ZN6Thread8callMainEPv + 66 (splunkd)
[0x00007F3A41373851] ? (/lib64/libpthread.so.0)
[0x00007F3A3FC3311D] clone + 109 (/lib64/libc.so.6)
Linux / saomb040 / 2.6.32-279.11.1.el6.x86_64 / #1 SMP Sat Sep 22 07:10:26 EDT 2012 / x86_64
Last few lines of stderr (may contain info on assertion failure, but also could be old):
splunkd: /opt/splunk/p4/splunk/branches/5.0.2/src/pipeline/input/FileInputTracker.cpp:229: static bool FileInputTracker::computeCRC(uint64_t*, FileDescriptor, const Str&, file_offset_t, file_offset_t): Assertion bytesToHash < 1048576' failed.
2013-02-15 08:49:32.034 -0500 splunkd started (build 149561)
2013-02-15 08:50:53.469 -0500 Interrupt signal received
2013-02-15 08:51:06.551 -0500 splunkd started (build 149561)
2013-02-15 08:52:13.072 -0500 Interrupt signal received
2013-02-15 08:52:23.736 -0500 splunkd started (build 149561)
splunkd: /opt/splunk/p4/splunk/branches/5.0.2/src/pipeline/input/FileInputTracker.cpp:229: static bool FileInputTracker::computeCRC(uint64_t*, FileDescriptor, const Str&, file_offset_t, file_offset_t): AssertionbytesToHash < 1048576' failed.

/etc/redhat-release: Red Hat Enterprise Linux Server release 6.3 (Santiago)
glibc version: 2.12
glibc release: stable
Threads running: 34
argv: [splunkd -p 9089 start]
terminating...

Splunk appears to start fine, then splunkd fails miserably with a horrible crash creating the above crashlog and the CRC error on bytesToHash

[splunkIT]

We have ran into a similar problem after upgrading to 5.0.2. It seemed to have been caused by one of the apps which was not up-to-dated, and not compatible with the latest splunk version. We started by moving most of the apps (except for the default ones provided during the splunk install) to a temp location, and was able to startup splunk.