Splunk on Mac crashing continually

[gedsays]

Hi,

I'm just installed Splunk 6.1 on a Mac with OS X 10.9.2, using splunk-6.1.0-206881-macosx-10.7-intel.dmg package and tried adding some data. Below is the crash report.

When I try to restart splunkd, it just crashes again after a little while with similar messages in the crash report. I also get these assertion failures and complaints about not being able to find the manifest file in the crash report:

2014-05-11 21:01:10.299 +1000 splunkd started (build 206881)
Assertion failed: (n <= rawSize()), function removeStartOfRaw, file /Users/eserv/wrangler/build-src/6.1.0/src/framework/PipelineData.h, line 382.
2014-05-11 22:09:08.649 +1000 splunkd started (build 206881)
Cannot open manifest file inside "/Applications/Splunk/var/lib/splunk/_internaldb/db/db_1398999122_1398999122_0/rawdata": No such file or directory
Cannot open manifest file inside "/Applications/Splunk/var/lib/splunk/_introspection/db/db_1399806670_1399806670_0/rawdata": No such file or directory
Assertion failed: (n <= rawSize()), function removeStartOfRaw, file /Users/eserv/wrangler/build-src/6.1.0/src/framework/PipelineData.h, line 382.

Any suggestions?

Thanks in advance,

Ged

[build 206881] 2014-05-11 21:33:09
Received fatal signal 6 (Abort trap: 6).
Cause:
Unknown signal origin (si_code=0).
Crashing thread: parsing
Registers:
RIP: [0x00007FFF8E3B3866] __pthread_kill + 10 (/usr/lib/system/libsystem_kernel.dylib)
RDI: [0x0000000000002D03]
RSI: [0x0000000000000006]
RBP: [0x000000010B4DFB80]
RSP: [0x000000010B4DFB58]
RAX: [0x0000000000000000]
RBX: [0x000000010B4E0000]
RCX: [0x000000010B4DFB58]
RDX: [0x0000000000000000]
R8: [0x00000000FFFFF000]
R9: [0x000000000000017E]
R10: [0x0000000008000000]
R11: [0x0000000000000206]
R12: [0x00000001068CBB58]
R13: [0x00000001068CBB6E]
R14: [0x0000000000000006]
R15: [0x00000001068C854C]
RFLAGS: [0x0000000000000206]
TRAPNO: [0x0000000000000085]
ERR: [0x0000000002000148]
CS: [0x0000000000000007]
GS: [0x00000000DD340000]
FS: [0x0000000000000000]

OS: OS/X
Arch: x86-64

Backtrace (NOTE: symbols may be wrong -- dladdr() is unreliable on OS/X):
[0x0000000107443A00] ?
[0x00007FFF8DFB5B1A] abort + 125 (/usr/lib/system/libsystem_c.dylib)
[0x00007FFF8DF7F98E] _assert_rtn + 272 (/usr/lib/system/libsystem_c.dylib)
[0x00000001058CE177] _ZN12PipelineData16removeStartOfRawEm + 183 (/Applications/Splunk/bin/splunkd)
[0x00000001058CEF0C] _ZN27StructuredDataHeaderRemoverD0Ev + 2684 (/Applications/Splunk/bin/splunkd)
[0x0000000105E9C2F7] _ZNSt8_Rb_treeI3StrSt4pairIKS0_12CronIntervalESt10_Select1stIS4_ESt4lessIS0_ESaIS4_EE4findERS2 + 9223 (/Applications/Splunk/bin/splunkd)
[0x00000001058CE9C2] _ZN27StructuredDataHeaderRemoverD0Ev + 1330 (/Applications/Splunk/bin/splunkd)
[0x00000001058CD928] _ZN6FifoFdC2ER8PathnameR15CowPipelineDataP9EventLoopP18FifoInputProcessorR13PropertiesMap + 8392 (/Applications/Splunk/bin/splunkd)
[0x0000000105C76327] _ZN22PersistentCacheVersionD0Ev + 11223 (/Applications/Splunk/bin/splunkd)
[0x0000000105F8F692] _ZN35TcpOutboundTerminateExternallyActorD0Ev + 6034 (/Applications/Splunk/bin/splunkd)
[0x00007FFF8C7BB899] _pthread_body + 138 (/usr/lib/system/libsystem_pthread.dylib)
[0x00007FFF8C7BB72A] _pthread_struct_init + 0 (/usr/lib/system/libsystem_pthread.dylib)
[0x00007FFF8C7BFFC9] thread_start + 13 (/usr/lib/system/libsystem_pthread.dylib

[gsteff]

I'm getting the error in Splunk 6.1 on Linux. It occurs when I add a new filesystem directory data input, and appears to relate to the contents of the files in it.

[lguinn2]

Hmm, I would try installing with the .tgz tar ball instead. I am not having trouble and that is what I used for my Mac. I'd also check permissions.

You should only start/stop Splunk as the same user who owns all the Splunk files. If you once started Splunk using root, then some of the file ownership may have changed. This is one way that permissions problems happen. You may want to use chown -R to fix that.

I'd also check that your download wasn't corrupt. But if all of these suggestions fail, I'd submit a bug: at that point, I would guess that Splunk published a bad download package for the Mac.

[gedsays]

OK. So I checked the md5sum against the .dmg package and they match. The chown -R wasn't necessary as the files are all owned by the right user.

So, I uninstalled and installed the .tgz tarball. I still get the same behaviour. After indexing the input files (about 1.5Gb worth - not sure if that is a problem given the 500Mb limit/day but it shouldn't crash the daemon), splunk crashed and when restarted, it crashes again within a few seconds. Errors are similar to my initial post.

Any help would be really appreciated!

[gedsays]

Thanks for the suggestions. I started Splunk with the same user as the one that ran the installation. The strange thing is that Splunk was running fine while adding a few million events from the input files and then at some point splunkd crashed and from then would crash almost immediately after restarting. I tried reinstalling Splunk and the same sequence of events occurred.

I'll check the md5sum of the installer, running chown -R and if that doesn't shed any light, will try the .tgz tarball install.