Monitoring Splunk

Why do I get this error "The system is approaching the maximum number of historical searches that can be run concurrently"?

ralopez
New Member

server have 4 processors and 16Gb of ram.
when this happens web applications starts to freeze.
rebooting the server does not solve this issue.
Disabling real time alerts did not work.

0 Karma

tiagofbmm
Influencer

Hey

The reason is that you are reaching your search quota, as the message says.

You may have scheduled searches and many users filling up your resources and making you reach that limit. Each role has specific search quotas for historical and real-time searches.

So each user that belongs to a role has its own set of limits for disk usage, historical searches, real time searches, and others you can find here http://docs.splunk.com/Documentation/Splunk/7.0.2/admin/authorizeconf#authorize.conf.example

If your limits.conf hasn't been changed, it means by default you have
max_hist_searches = max_searches_per_cpu x number_of_cpus + base_max_searches
Which would be 1x4+6=10
The same number of real-time searches.

You can check all those parameters in http://docs.splunk.com/Documentation/Splunk/7.0.2/Admin/Limitsconf

You may need to disable some scheduled searches to get this controlled actually. Check your scheduled saved searches and start disabling them.

0 Karma

ralopez
New Member

hey,

Thanks for the information, but sadly it did not work for me.

When i check the logs to see if i could find a specific error for my case, in the splunkd.log, i found the following errors:
03-21-2018 06:43:35.005 -0400 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\splunk_app_db_connect\bin\mi_input.py"" self.stream.flush()
03-21-2018 06:43:35.005 -0400 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\splunk_app_db_connect\bin\mi_input.py"" IOError: [Errno 22] Invalid argument
03-21-2018 06:43:35.005 -0400 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\splunk_app_db_connect\bin\mi_input.py"" Logged from file None, line None
03-21-2018 06:43:35.348 -0400 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\splunk_app_db_connect\bin\mi_input.py"" Degrade mode - ENTERING - (pid=4124) rename failed. File in use?

This errors are from the splunk dbconnect app, but i do not know what they mean.

Does this have something to do with the maximum number of historical search?

0 Karma

ralopez
New Member

And cannot run any search.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...