server have 4 processors and 16Gb of ram.
when this happens web applications starts to freeze.
rebooting the server does not solve this issue.
Disabling real time alerts did not work.
Hey
The reason is that you are reaching your search quota, as the message says.
You may have scheduled searches and many users filling up your resources and making you reach that limit. Each role has specific search quotas for historical and real-time searches.
So each user that belongs to a role has its own set of limits for disk usage, historical searches, real time searches, and others you can find here http://docs.splunk.com/Documentation/Splunk/7.0.2/admin/authorizeconf#authorize.conf.example
If your limits.conf hasn't been changed, it means by default you have
max_hist_searches = max_searches_per_cpu x number_of_cpus + base_max_searches
Which would be 1x4+6=10
The same number of real-time searches.
You can check all those parameters in http://docs.splunk.com/Documentation/Splunk/7.0.2/Admin/Limitsconf
You may need to disable some scheduled searches to get this controlled actually. Check your scheduled saved searches and start disabling them.
hey,
Thanks for the information, but sadly it did not work for me.
When i check the logs to see if i could find a specific error for my case, in the splunkd.log, i found the following errors:
03-21-2018 06:43:35.005 -0400 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\splunk_app_db_connect\bin\mi_input.py"" self.stream.flush()
03-21-2018 06:43:35.005 -0400 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\splunk_app_db_connect\bin\mi_input.py"" IOError: [Errno 22] Invalid argument
03-21-2018 06:43:35.005 -0400 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\splunk_app_db_connect\bin\mi_input.py"" Logged from file None, line None
03-21-2018 06:43:35.348 -0400 ERROR ExecProcessor - message from "python "C:\Program Files\Splunk\etc\apps\splunk_app_db_connect\bin\mi_input.py"" Degrade mode - ENTERING - (pid=4124) rename failed. File in use?
This errors are from the splunk dbconnect app, but i do not know what they mean.
Does this have something to do with the maximum number of historical search?
And cannot run any search.