Splunk Crashing once in 10min

[KarunK]

Hi All,

My Splunk instance 5.0.1 running in Solaris 10 is crashing. I have updated with the latest Splunk 5.0.3 but made it worse. It used generate crash logs and crash files (in /var/core directory) once in every 10 min. Not its twice in 10 min.

Can anyone help ?

Crash log is added below. Looks like the Report Acceleration is causing the crash.

Thanks

Kind Regards

KK

bash-3.2# more crash-2013-06-27-15:13:19.log
[build 163460] 2013-06-27 15:13:19
Received fatal signal 6 (Abort).
 Cause:
   Unknown signal origin (si_code=-1).
 Crashing thread: dispatch
 Registers:
    RIP:  [0xFFFFFD7FFEAE2CEA] __lwp_kill + 10 (/lib/amd64/libc.so.1)
    RDI:  [0x0000000000000003]
    RSI:  [0x0000000000000006]
    RBP:  [0xFFFFFD7FFE3FD3A0]
    RSP:  [0xFFFFFD7FFE3FD398]
    RAX:  [0x0000000000000000]
    RBX:  [0x0000000000000006]
    RCX:  [0x0000000000000005]
    RDX:  [0xFFFFFFFF83986C80]
    R8:  [0x000000000000002D]
    R9:  [0x0000000000000000]
    R10:  [0x0000000000000005]
    R11:  [0x0000000000000000]
    R12:  [0x0000000002CDB1B8]
    R13:  [0x0000000002CDB010]
    R14:  [0x0000000002CDB1E8]
    R15:  [0x0000000002CB9210]
    RFL:  [0x0000000000000286]
    TRAPNO:  [0x000000000000000E]
    ERR:  [0x0000000000000014]
    CS:  [0x000000000000004B]
    GS:  [0x0000000000000000]
    FS:  [0x0000000000000000]

 OS: SunOS
 Arch: x86-64

 Backtrace:
  [0xFFFFFFFFFFFFFFFF] ?
  [0xFFFFFD7FFEA87E99] raise + 25 (/lib/amd64/libc.so.1)
  [0xFFFFFD7FFEA6694E] abort + 94 (/lib/amd64/libc.so.1)
  [0x0000000001A0161F] _ZN9__gnu_cxx27__verbose_terminate_handlerEv + 351 (/opt/splunk/bin/splunkd)
  [0x0000000001A002A6] _ZN10__cxxabiv111__terminateEPFvvE + 6 (/opt/splunk/bin/splunkd)
  [0x0000000001A002D3] _ZSt9terminatev + 19 (/opt/splunk/bin/splunkd)
  [0x0000000001A0065F] __cxa_pure_virtual + 31 (/opt/splunk/bin/splunkd)
  [0x0000000000D1E73E] _ZN15SearchEvaluator10lispyQueryER3StrR7TimevalS3_R9StrVectorRKS2_S7_b + 414 (/opt/splunk/bin/splunkd)
  [0x0000000000B3644C] _ZN17IndexScopedSearch4initERK7TimevalS2_bP14LookupOperatorP12FieldAliaserP18CalcFieldProcessorPKSt3setI10CMBucketIdSt4lessISA_ESaISA_EE + 588 (/opt/splunk/bin/splunkd)
  [0x0000000000B26C9E] _ZN14SearchOperator8evalArgsER17SearchResultsInfo + 9006 (/opt/splunk/bin/splunkd)
  [0x0000000000FA901A] _ZN14SearchPipeline8evalArgsER17SearchResultsInfo + 90 (/opt/splunk/bin/splunkd)
  [0x0000000000B572B9] _ZN22BucketSummaryProcessor8evalArgsER17SearchResultsInfo + 8713 (/opt/splunk/bin/splunkd)
  [0x0000000000FA901A] _ZN14SearchPipeline8evalArgsER17SearchResultsInfo + 90 (/opt/splunk/bin/splunkd)
  [0x000000000103B120] _ZN14DispatchThread8evaluateEbb + 15264 (/opt/splunk/bin/splunkd)
  [0x0000000001033981] _ZN14DispatchThread8mainImplEv + 4321 (/opt/splunk/bin/splunkd)
  [0x00000000010368C2] _ZN14DispatchThread4mainEv + 226 (/opt/splunk/bin/splunkd)
  [0x0000000000F37352] _ZN6Thread8callMainEPv + 98 (/opt/splunk/bin/splunkd)
  [0xFFFFFD7FFEADD1AB] _thr_slot_offset + 795 (/lib/amd64/libc.so.1)
  [0xFFFFFD7FFEADD3E0] smt_pause + 96 (/lib/amd64/libc.so.1)
 SunOS / splunk / 5.10 / Generic_147441-07 / i86pc
 Last few lines of stderr (may contain info on assertion failure, but also could be old):
    2013-06-26 17:19:51.400 +1000 splunkd started (build 143156)
    2013-06-26 17:25:11.350 +1000 Interrupt signal received
    2013-06-26 17:27:59.775 +1000 splunkd started (build 143156)
    2013-06-27 12:21:03.153 +1000 Interrupt signal received
    2013-06-27 12:21:56.892 +1000 splunkd started (build 143156)
    2013-06-27 13:21:08.304 +1000 Interrupt signal received
    2013-06-27 13:37:12.340 +1000 splunkd started (build 163460)
    2013-06-27 13:39:12.006 +1000 Interrupt signal received
    2013-06-27 13:39:59.495 +1000 splunkd started (build 163460)
    2013-06-27 13:52:08.211 +1000 Interrupt signal received
    2013-06-27 13:52:58.376 +1000 splunkd started (build 163460)
    2013-06-27 14:50:25.221 +1000 Interrupt signal received
    2013-06-27 15:04:15.911 +1000 splunkd started (build 163460)

Threads running: 3
argv: [splunkd -p 8089 start]
Process renamed: [splunkd pid=3972] splunkd -p 8089 start [process-runner]
Process renamed: [splunkd pid=3972] search --id=SummaryDirector_1372309985.40 --maxbuckets=0 --ttl=30 --maxout=50000 --maxtime=8640000 --lookups=0 --reduce_freq=10 --user=splunk-system-user --pro --roles=admin:can_delete:cds:power:splunk
-system-role:user
terminating...
bash-3.2#

[MuS]

Hi KarunK

open files is too low, check the docs about ulimit:

http://docs.splunk.com/Documentation/Splunk/5.0.3/Installation/Systemrequirements
..snip..
Usually, the default file descriptor limit (ulimit) on a *nix-based OS is 1024. Your Splunk administrator should determine the correct level, but it should be at least 8192.
..schnapp..

If this does not help do as kristian told you, make a diag and file a support case.

cheers, MuS

[KarunK]

Memory and CPU look good."splukd.log" have a lot of entries like below " DispatchCommand - could not read metadata file: /opt/splunk/var/run/splunk/dispatch/rt_scheduler_nobodycds_RMD5e57c4bb343ae7e10_at_1372658189_0.13518/metadata.csv"

[KarunK]

core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
open files (-n) 256
pipe size (512 bytes, -p) 10
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 27605
virtual memory (kbytes, -v) unlimited

[kristian_kolb]

Check your ulimit for open files, and make a diag-dump and open a support case.

[MHibbin]

Is there anything of interest in $SPLUNK_HOME\var\log\splunk\splunkd.log?

Additionally what's the situation with memory/CPU utilisation?