Splunk Search

maxsearches limit reached

kamlesh_vaghela
SplunkTrust
SplunkTrust

Hi ,
I have configured 20 Alerts with below run every - "5 min" and Alert mode "Once Per Result". Each Alerts are generating more then 1000 events . As the Alert Action , a shell script is called.
So with this , Some of the alerts are executed properly and others are skipped. See below log from "scheduler.log".

12-23-2015 17:44:58.494 +0530 INFO SavedSplunker - savedsearch_id="admin;search;Alert 20", user="admin", app="search", savedsearch_name="Alert 20", status=skipped, reason="maxsearches limit reached", scheduled_time=1450872600, window_time=0

Kindly suggest me what should I do in this kind of situations.

Thanks

Tags (1)
1 Solution

jkat54
SplunkTrust
SplunkTrust

Please read the upvoted answer on this question:

https://answers.splunk.com/answers/297030/what-is-the-max-value-for-maxsearches-is-there-a-w.html

Short answer: this can be adjusted in limits.conf.
Warning: limits.conf must be on peers and search heads for this setting to work in distributed environment

Long answer: why not stagger your searches so that you're not executing 20 at the exact same time.?

*/5 * * * *   <- every five on the hour
1-59/5 * * * * <- every five starting at 1 minute after the hour
2-59/5 * * * * <- every five starting at 2 minutes after the hour

View solution in original post

jkat54
SplunkTrust
SplunkTrust

Please read the upvoted answer on this question:

https://answers.splunk.com/answers/297030/what-is-the-max-value-for-maxsearches-is-there-a-w.html

Short answer: this can be adjusted in limits.conf.
Warning: limits.conf must be on peers and search heads for this setting to work in distributed environment

Long answer: why not stagger your searches so that you're not executing 20 at the exact same time.?

*/5 * * * *   <- every five on the hour
1-59/5 * * * * <- every five starting at 1 minute after the hour
2-59/5 * * * * <- every five starting at 2 minutes after the hour

peter_krammer
Communicator

Which limit in limits.conf do I need to change to allow the search to be queued?
We have alerts that we definitely do not want to be skipped, but sometimes they take longer to execute and the following search is skipped because the first is not finished yet.

0 Karma

biec1
Explorer

The following is the current configuration in our environment. We are getting max_concurrent limit reached on one of the three Search Heads for specific Saved Search. It does not happen on rest of the two Search Heads.
Search Head:- 3x16+10=58
Search Peer :- 1x20+6=26

Is it necessary to maintain identical parameters on both Search peers and Search heads?
Shall i increase max_searches_per_cpu on Search Peers from 1 to 2?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...