Why is my Splunk instance not starting and crashing with crash logs?

[sylim_splunk]

My splunk instance keeps on crashing it's not even starting. Whenever "splunk start" is entered it creates crash logs under /var/log/splunk directory, which has below stack traces with "Aborted" and assertion, _parent == __null' failed
This started to happen after upgrade to 7.0.4 from 6.5.5 which did not show any crashes.
*

[build c8a78efdd40f] 2017-11-30 09:49:09
 Received fatal signal 6 (Aborted).
 Cause:
 Signal sent by PID 15459 running under UID 11396.
 Crashing thread: HttpInputServerManagementThread
 Registers:
 RIP: [0x00007FED9E034495] gsignal + 53 (libc.so.6 + 0x32495)

OS: Linux
 Arch: x86-64
Backtrace (PIC build):
 [0x00007FED9E034495] gsignal + 53 (libc.so.6 + 0x32495)
 [0x00007FED9E035C75] abort + 373 (libc.so.6 + 0x33C75)
 [0x00007FED9E02D60E] ? (libc.so.6 + 0x2B60E)
 [0x00007FED9E02D6D0] __assert_perror_fail + 0 (libc.so.6 + 0x2B6D0)
 [0x00007FEDA0CD31AA] ? (splunkd + 0x16C61AA)
 [0x00007FEDA0CD48E8] _ZN13HttpInputConf4loadEv + 1384 (splunkd + 0x16C78E8)
 [0x00007FEDA0CD51EE] _ZN13HttpInputConfC2Ev + 94 (splunkd + 0x16C81EE)
 [0x00007FEDA0CD5251] _ZN13HttpInputConf6Getter6updateEv + 33 (splunkd + 0x16C8251)
 [0x00007FEDA0CCAE18] _ZN31HttpInputServerManagementThread11reconfigureEbR3Str + 120 (splunkd + 0x16BDE18)
 [0x00007FED9FE57B6C] _ZN31HttpInputServerManagementThread4mainEv + 172 (splunkd + 0x84AB6C)
 [0x00007FEDA090FB3F] _ZN6Thread8callMainEPv + 111 (splunkd + 0x1302B3F)
 [0x00007FED9E39DAA1] ? (libpthread.so.0 + 0x7AA1)
 [0x00007FED9E0EABCD] clone + 109 (libc.so.6 + 0xE8BCD)

 splunkd: /home/build/build-src/minty/src/pipeline/input/httpinput/HttpInputConf.cpp:111: void HttpInputConf::TokenConf::setParent(const HttpInputConf::TokenConf*): **Assertion \`_parent == __null' failed.**
 splunkd: /home/build/build-src/minty/src/pipeline/input/httpinput/HttpInputConf.cpp:111: void HttpInputConf::TokenConf::setParent(const HttpInputConf::TokenConf*): **Assertion \`_parent == __null' failed**.
*

Please help.

[sylim_splunk · Accepted solution]

This can happen when token for Http Event collector is used multiple times in different "http" stanzas. You can confirm if it is the case or not by running the below;

$SPLUNK_HOME/bin/splunk btool ---debug inputs list | grep "token =" |cut -d "=" -f 2 | sort |uniq -c | awk '{if ($1>1) print $1,$2 }'

If you find tokens used more than 1 it can cause crash. Make sure the tokens are only used once. We are implementing fixes so that it puts ERROR messages when it detects duplicate tokens used for HEC and continue, no more crashes.