Splunk Search

How to increase the maximum number of real-time searches

Bliide
Path Finder

I am trying to do a stress test on a new server in a fresh Splunk environment. I would like to increase the number of real-time searches allowed and see how much of a performance hit the server takes. I know I need to edit limits.conf, but I am not certain what stanza to add and what values to use.

Anyone with experience in tweaking limits.conf is what I am looking for. Is it best to start off by just adding a search stanza with: max_rt_search_multiplier = 2

or is it better to add:

max_searches_per_cpu = 2 ?

Our goal to is see how many concurrent real-time searches we can run before we start having a substantial performance hit.

0 Karma

steveyz
Splunk Employee
Splunk Employee

changing the max_rt_search_multiplier is the way to go. changing max_searches_per_cpu and base_max_searches will change also affect the real-time limit, but will alter the limit for historical searches too.

max real-time searches = max_rt_search_multiplier x (max_searches_per_cpu * + base_max_searches)

And to address woodcock's comment, there is a setting that you can tweak which will allow you to optionally trade performance for latency. It will run real-time searches with higher latency but generally use far less system resources. The setting is under

[realtime]
indexed_realtime_use_by_default = true/false (defaults to false. set to true for less resource usage but higher latency)

woodcock
Esteemed Legend

I can tell you that answer: ONE! Unless you have designated your entire cluster to the purpose of running Real-Time Searches, don't run any.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...