Getting Data In

typing queue always full with indexer crash

surekhasplunk
Communicator

Hi

Once my indexer crashed with below error:
kernel: splunkd[] general protection ip:xyz error:0 in splunkd[]

And after restarting the indexer my Parsing, Merging , Typing queues are always full with going to the indexing queue.

How to resolve this issue ?

Tags (2)
0 Karma
1 Solution

PavelP
Motivator

General protection error is a kernel message, I suspect it is not a configuration issue. You have to open a support ticket. To confirm it:

  • check logs for similar errors with "grep kernel | grep error" - do you see other errors? are all of them related to splunk or there are other processes?
  • check for crash* logs in $SPLUNK_HOME/var/log/splunk
  • check /var/log/messages or journal for OOM or similar OS errors

View solution in original post

0 Karma

PavelP
Motivator

General protection error is a kernel message, I suspect it is not a configuration issue. You have to open a support ticket. To confirm it:

  • check logs for similar errors with "grep kernel | grep error" - do you see other errors? are all of them related to splunk or there are other processes?
  • check for crash* logs in $SPLUNK_HOME/var/log/splunk
  • check /var/log/messages or journal for OOM or similar OS errors
0 Karma

surekhasplunk
Communicator

Hi @PaveIP

I raised case with support but they are going in completely different direction all together instead of finding the root cause of this particular problem they are like change the whole setup upgrade or move conf files etc.
We can do that but not now. Not at that time when the crash happened but at a later point.

1) And i did check log for kernel or error and found something like below
16:00:53 SPKLX kernel: device bond0 left promiscuous mode
16:00:53 SPKLX kernel: device em1 left promiscuous mode
16:00:53 SPKLX kernel: device bond0 entered promiscuous mode
16:00:53 SPKLX kernel: device em1 entered promiscuous mode

2) i checked for crash.log and they have :
Received fatal signal 11 (Segmentation fault).
Cause:
Unknown signal origin (si_code=128, si_addr=[0x0000000000000000]).
Crashing thread: CallbackRunnerThread
Registers:
RIP: [0x00007F719784F505]

3)i cant see any OOM or OS errors.

0 Karma

PavelP
Motivator

thank you @surekhasplunk for an update

1) information messages related to tcpdump
2) if all crashlogs have "CallbackRunnerThread" as a cause it could be splunk fault and an update can help
3) -

good luck with troubleshooting!

0 Karma

anmolpatel
Builder

@surekhasplunk Potential things to check:
- the transforms that are applied to the sourcetypes, there could be issues with that if you've created custom transforms or routing of the data
- SED extraction in props.conf

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...