Getting Data In

How do I find what is causing my typing queue blockage?

hrawat_splunk
Splunk Employee
Splunk Employee

How do I find sources/source types/hosts/indexes causing typing queue blockage?

Tags (2)
1 Solution

hrawat_splunk
Splunk Employee
Splunk Employee

Steps
1) Set under [default] stanza in limits.conf

regex_cpu_profiling = true

regex_cpu_profiling =
* Enable CPU time metrics for RegexProcessor. Output will be in the
metrics.log file.
Entries in metrics.log will appear per_host_regex_cpu, per_source_regex_cpu,
per_sourcetype_regex_cpu, per_index_regex_cpu.
* Default: false

2) Set under [metrics] stanza in limits.conf

maxseries = 50

maxseries =
* The number of series to include in the per_x_thruput reports in metrics.log.
* Default: 10

3) restart splunk

4) Wait for typing queue to block.

5) Goto splunk UI and following queries will be helpful:

Which source type is taking most of the cpu time.

index=_internal host= source=*metrics.log group=per_sourcetype_regex_cpu |timechart max(cpu) by series

Which source type is taking most of the cpu time per event:

index=_internal host= source=*metrics.log group=per_sourcetype_regex_cpu |timechart max(cpupe) by series

Repeat queries for per_host_regex_cpu, per_source_regex_cpu, and per_index_regex_cpu(if needed)

cpu > total cpu time for a given series
cpupe > cpu time per event for a given series
bytes > total bytes processes for a given series
ev > total events for a given series

View solution in original post

hrawat_splunk
Splunk Employee
Splunk Employee

Steps
1) Set under [default] stanza in limits.conf

regex_cpu_profiling = true

regex_cpu_profiling =
* Enable CPU time metrics for RegexProcessor. Output will be in the
metrics.log file.
Entries in metrics.log will appear per_host_regex_cpu, per_source_regex_cpu,
per_sourcetype_regex_cpu, per_index_regex_cpu.
* Default: false

2) Set under [metrics] stanza in limits.conf

maxseries = 50

maxseries =
* The number of series to include in the per_x_thruput reports in metrics.log.
* Default: 10

3) restart splunk

4) Wait for typing queue to block.

5) Goto splunk UI and following queries will be helpful:

Which source type is taking most of the cpu time.

index=_internal host= source=*metrics.log group=per_sourcetype_regex_cpu |timechart max(cpu) by series

Which source type is taking most of the cpu time per event:

index=_internal host= source=*metrics.log group=per_sourcetype_regex_cpu |timechart max(cpupe) by series

Repeat queries for per_host_regex_cpu, per_source_regex_cpu, and per_index_regex_cpu(if needed)

cpu > total cpu time for a given series
cpupe > cpu time per event for a given series
bytes > total bytes processes for a given series
ev > total events for a given series

dshpritz
SplunkTrust
SplunkTrust

This is a fantastic post. The only thing I would add is that the regex_cpu_profiling was added in 6.6. Thanks!

hrawat_splunk
Splunk Employee
Splunk Employee

It's integrated with DMC as well starting 7.x. However enabling regex_cpu_profiling is required.

dshpritz
SplunkTrust
SplunkTrust

Didn't know that. This post is solid gold!

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...