How to troubleshoot why the Splunk 7.1.x search head crashes periodically?

[khusain_splunk]

Splunk search head crashes periodically, Splunk version is 7.1. Below is the crash logs:

Received fatal signal 6 (Aborted).
Cause:
 Signal sent by PID 14764 running under UID 2000.
Crashing thread: SHPHeartbeatThread
Registers:
RIP: xxxxxxxxxxxxxxxxxxxxxxxx
  RDI:  xxxxxxxxxxxxxxxxxxxxxxxxx
  RSI: xxxxxxxxxxxxxxxxxxxxxxxx
  RBP:  xxxxxxxxxxxxxxxxxxxxxxx
  RSP:  xxxxxxxxxxxxxxxxxxxxxx
  RAX:  xxxxxxxxxxxxxxxxxxxxxxxx

ERROR - Failed opening "": No such file or directory
  Conf mutator lockfile has disappeared; error condition possible.
  Bypassing local license checks since this instance is configured with a remote license master.
  xxxxxxxxx 12:07:25.476 +0800 splunkd started (build 8f0ead9ec3db)
  splunkd: /home/build/build-src/nightlight/src/framework/SearchResultsMem.cpp:839: SearchResultsMem::iterator SearchResultsMem::erase(SearchResultsMem::iterator, SearchResultsMem::iterator): Assertion `it != end()' failed.
xxxxxxxxxxxx 10:46:31.839 +0800 splunkd started (build 8f0ead9ec3db)
  splunkd: /home/build/build-src/nightlight/src/framework/SearchResultsMem.cpp:839: SearchResultsMem::iterator SearchResultsMem::erase(SearchResultsMem::iterator, SearchResultsMem::iterator): Assertion `it != end()' failed.

[khusain_splunk · Accepted solution]

Hi,

This is a know bug in Splunk version 7.1. If below is the reason for crash then it is a code bug and already fixed in Splunk 7.2.

splunkd: /home/build/build-src/nightlight/src/framework/SearchResultsMem.cpp:839: SearchResultsMem::iterator SearchResultsMem::erase(SearchResultsMem::iterator, SearchResultsMem::iterator): Assertion `it != end()' failed.

[josephliion]

Do you know what is the Issue number of this bug?

[dwaddle]

1. Look for that assertion in any known issues in the splunk docs
2. Upgrade to the latest patch level in your version (if you are in Splunk 7.1 release train, then 7.1.x where x is the most recent)
3. Submit a support case

When splunk exits on an assertion failure, this is it saying that a internal constraint in the code has been violated. Usually, these require access to the source code to understand and require a code fix to resolve. If it's not been resolved already, then you will need a support case.