Getting Data In

How to troubleshoot why Windows event forwarders are reporting typing and parsingqueue blocked messages, causing delayed forwarding and indexing?

a212830
Champion

Hi,

I've had complaints from customers that data is taking too long to appear in the system. Today, one of the Windows event forwarders was 2 hours behind... I looked for "blocked" messages, and see lots of typing queue and parsingqueue blocked messages. How can I troubleshoot these? Are there any settings that can be configured to help? (I'm at Splunk 6.1.9).

0 Karma

jkat54
SplunkTrust
SplunkTrust

Sometimes this happens when your CPU or Memory Usage is too high on the forwarders, or even the indexers.

Check your CPU usage and load averages, etc. If they are too high, you may need to upgrade some systems.

Or maybe even you need to optimize some large searches, etc.

A common cause of this is monitoring a super large folder like this:

[monitor://path/.../*.log]

You're effectively telling splunk to index everything under /path thats a .log file but maybe there are billions of files in this directory that are .txt files, etc. Splunk has to keep a running list of what it's already processed, etc. and things get messy... especially on non-reference hardware such as 1 CPU virtual machines, etc.

0 Karma

a212830
Champion

Is this indicating a problem on the forwarders, or the indexers? I don't really see any way to determine that...

My indexers are loaded with resources - 48 cpu's and 256gb of memory, so I'd be very surprised if that's the issue. This particular forwarder processes event logs, at about 7 - 10k per minute.

0 Karma
Get Updates on the Splunk Community!

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Wednesday, May 29, 2024  |  11AM PST / 2PM ESTRegister now and join us to learn more about how you can ...

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer Certification at ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...

Share Your Ideas & Meet the Lantern team at .Conf! Plus All of This Month’s New ...

Splunk Lantern is Splunk’s customer success center that provides advice from Splunk experts on valuable data ...