Splunk Search

How to increase the maximum number of real-time searches

Bliide
Path Finder

I am trying to do a stress test on a new server in a fresh Splunk environment. I would like to increase the number of real-time searches allowed and see how much of a performance hit the server takes. I know I need to edit limits.conf, but I am not certain what stanza to add and what values to use.

Anyone with experience in tweaking limits.conf is what I am looking for. Is it best to start off by just adding a search stanza with: max_rt_search_multiplier = 2

or is it better to add:

max_searches_per_cpu = 2 ?

Our goal to is see how many concurrent real-time searches we can run before we start having a substantial performance hit.

0 Karma

steveyz
Splunk Employee
Splunk Employee

changing the max_rt_search_multiplier is the way to go. changing max_searches_per_cpu and base_max_searches will change also affect the real-time limit, but will alter the limit for historical searches too.

max real-time searches = max_rt_search_multiplier x (max_searches_per_cpu * + base_max_searches)

And to address woodcock's comment, there is a setting that you can tweak which will allow you to optionally trade performance for latency. It will run real-time searches with higher latency but generally use far less system resources. The setting is under

[realtime]
indexed_realtime_use_by_default = true/false (defaults to false. set to true for less resource usage but higher latency)

woodcock
Esteemed Legend

I can tell you that answer: ONE! Unless you have designated your entire cluster to the purpose of running Real-Time Searches, don't run any.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...