Getting Data In

Troubleshooting Splunk Queues (Typing Queue)

mbrunetto
Path Finder

My Typing Queue is currently blocking and causing backups. I believe I have the order right
udpin/splunktcpin, parsing, and agg queues are all backed up. Indexing queue has some localized spikes, but is mostly at 0. This should indicate a delay in the Typing Pipeline. My data comes in waves with the workday, and the queues max during the workday, and clear out overnight.

Where would I go next to try and clear these queues out? What are my troubleshooting steps? It looks like this pipeline is trying to do regex's and punctuation; but how do I see what part of the pipeline is holding up the queue? I'd like to find out if it's something that I've put in, and if so, which thing to remove.

Since the index seems unblocked, I don't think this has anything to do with my disk speed. My CPUs (8) are busy, but not overworked, and I have plenty of free memory. I run a single box doing indexer/search on 10G of data/day.

phoffman_splunk
Splunk Employee
Splunk Employee

1st easiest thing to start with is to download and install the S.o.S app (app link here) If you install this on your search head, remember to deploy the TA (Links here on the documentation tab) to your indexer(s).

In the S.o.S. app, check out the "Estimated percentage of total CPU used per Splunk processor" panel under the "Indexing Performance" dashboard. This will let you view where most of your CPU processing time is going. most typically it is a bad regex.

Then it is a matter of finding the bad regex that was put in place, through exploring your transorms.conf settings through the S.o.S. "Configuration File Viewer" view.

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...