Backtrace:
[0x000000391D032625] gsignal + 53 (/lib64/libc.so.6)
[0x000000391D033E05] abort + 373 (/lib64/libc.so.6)
[0x000000391D02B74E] ? (/lib64/libc.so.6)
[0x000000391D02B810] __assert_perror_fail + 0 (/lib64/libc.so.6)
[0x000000000099145A] ? (splunkd)
[0x000000000098D582] _ZNK11TailWatcher12setupConfigsER15WatchedTailFile + 1474 (splunkd)
[0x000000000098D692] _ZNK11TailWatcher19initializeFileStateER15WatchedTailFileRK8Pathname + 66 (splunkd)
[0x00000000009904B2] _ZN11TailWatcher11fileChangedEP16WatchedFileStateRK7Timeval + 242 (splunkd)
[0x0000000000EC2602] _ZN30FilesystemChangeInternalWorker15callFileChangedER7TimevalP16WatchedFileState + 114 (splunkd)
[0x0000000000EC3F90] _ZN30FilesystemChangeInternalWorker12when_expiredERy + 464 (splunkd)
[0x0000000000F53B2D] _ZN11TimeoutHeap18runExpiredTimeoutsER7Timeval + 301 (splunkd)
[0x0000000000EBD818] _ZN9EventLoop3runEv + 744 (splunkd)
[0x000000000098E9ED] _ZN11TailWatcher3runEv + 141 (splunkd)
[0x000000000099428A] _ZN13TailingThread4mainEv + 154 (splunkd)
[0x0000000000F5165E] _ZN6Thread8callMainEPv + 62 (splunkd)
[0x000000391D8079D1] ? (/lib64/libpthread.so.0)
[0x000000391D0E88FD] clone + 109 (/lib64/libc.so.6)
Linux / dggtsp104-or / 2.6.32-431.20.3.el6.x86_64 / #1 SMP Fri Jun 6 18:30:54 EDT 2014 / x86_64
Last few lines of stderr (may contain info on assertion failure, but also could be old):
2015-07-22 15:39:57.112 +0800 splunkd started (build 271043)
2015-07-22 15:41:07.464 +0800 Interrupt signal received
2015-07-22 15:41:13.389 +0800 splunkd started (build 271043)
Conf mutator lockfile has disappeared; error condition possible.
2015-07-22 15:46:30.551 +0800 splunkd started (build 271043)
Conf mutator lockfile has disappeared; error condition possible.
2015-07-22 15:50:47.434 +0800 splunkd started (build 271043)
splunkd: /home/build/build-src/6.2.4/src/pipeline/input/Tailing.h:120: bool StatWrap::isDir() const: Assertion `_valid' failed.
/etc/redhat-release: Red Hat Enterprise Linux Server release 6.5 (Santiago)
glibc version: 2.12
glibc release: stable
Last errno: 2
Threads running: 31
argv: [splunkd -p 8089 restart]
Thread: "MainTailingThread", did_join=0, ready_to_run=Y, main_thread=N
First 8 bytes of Thread token @0x7fd98c81e150:
00000000 00 f7 7f 8c d9 7f 00 00 |........|
00000008
Hi,
This is related to a big introduced in 6.2.4, the fix is included in 6.2.5 which is available for download
2015-8-11 SPL-104017 Splunkd crash due to assertion failure in Tailing.