Splunk Search

Streamed search execute failed because: Error in 'surrounding': Too many events (> 10000) in a single second.

splunk_zen
Builder

Even though I have overwritten what I believe is this limit in limits.conf,
btool is showing,

[show_source]
max_count = 50000
distributed_search_limit = 30000
distributed = true

The error message displays 10k rather than 50k.
Is this a bug as in the parameter is not being respected, or a bug as in the message not displaying the value Splunk is enforcing ?

Any recommendation on how to allow to check the source for

0 Karma
1 Solution

jkat54
SplunkTrust
SplunkTrust

1st, know your limits:

http://docs.splunk.com/Documentation/Splunk/6.2.3/admin/Limitsconf

limits.conf settings and DISTRIBUTED SEARCH
Unlike most settings which affect searches, limits.conf settings are not
provided by the search head to be used by the search peers.
This means that if
you need to alter search-affecting limits in a distributed environment, typically
you will need to modify these settings on the relevant peers and search head for
consistent results.

2nd, tell us your architecture. If you only have 1 server, my answer above is null and void.

3rd, as mentioned there is a configuration file precedence issue possibly. See the following:
http://docs.splunk.com/Documentation/Splunk/6.2.0/admin/Wheretofindtheconfigurationfiles

View solution in original post

jkat54
SplunkTrust
SplunkTrust

1st, know your limits:

http://docs.splunk.com/Documentation/Splunk/6.2.3/admin/Limitsconf

limits.conf settings and DISTRIBUTED SEARCH
Unlike most settings which affect searches, limits.conf settings are not
provided by the search head to be used by the search peers.
This means that if
you need to alter search-affecting limits in a distributed environment, typically
you will need to modify these settings on the relevant peers and search head for
consistent results.

2nd, tell us your architecture. If you only have 1 server, my answer above is null and void.

3rd, as mentioned there is a configuration file precedence issue possibly. See the following:
http://docs.splunk.com/Documentation/Splunk/6.2.0/admin/Wheretofindtheconfigurationfiles

splunk_zen
Builder

Thanks jkat54.
That makes sense, will push these changes to the Indexer cluster then.
There's no configuration file precedence issue as confirmed by btool, set this up in a specific App to exclusively target limits.conf (thus taking predence over system/{default,local})

jkat54
SplunkTrust
SplunkTrust

number 1 should help you then! Thanks for marking the answer.

0 Karma

renjith_nair
SplunkTrust
SplunkTrust

Try to do that in local/limits.conf and restart splunk after that if not done already

Happy Splunking!
0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...