If the system-wide real-time search limit is reached, can users still run regular searches, or will all searches at that point start being queued until a real-time search is closed?
If the max_rt_searches limit is reached you will be at your system-wide limit (max_hist_searches) and ad-hoc searches will be queued and scheduled searches will be skipped or continued (you can read more about the real time scheduling mode vs the continuous scheduling mode here: http://docs.splunk.com/Documentation/SplunkCloud/6.6.0/Report/Configurethepriorityofscheduledreports...)
with an example of a single instance with 12 cpu cores:
max_hist_searches = max_searches_per_cpu x number_of_cpus + base_max_searches
max_hist_searches = 1 x 12 + 6 = 18
max_rt_searches = max_rt_search_multiplier x max_hist_searches
max_rt_searches = 1 x 18
These settings live in limits.conf
http://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf?ac=partner_smt
base_max_searches =
A constant to add to the maximum number of searches, computed as a
multiplier of the CPUs.
Default: 6
max_rt_search_multiplier =
A number by which the maximum number of historical searches is multiplied
to determine the maximum number of concurrent real-time searches.
Note: The maximum number of real-time searches is computed as:
max_rt_searches = max_rt_search_multiplier x max_hist_searches
Default: 1
max_searches_per_cpu =
The maximum number of concurrent historical searches for each CPU.
The system-wide limit of historical searches is computed as:
max_hist_searches = max_searches_per_cpu x number_of_cpus + base_max_searches
NOTE: The maximum number of real-time searches is computed as:
max_rt_searches = max_rt_search_multiplier x max_hist_searches
Default: 1