Monitoring Splunk

Impact of real-time distributed searches on CPU utilization?

gregbujak
Path Finder

What is the impact of running real-time searches across a Splunk cluster, both for the dedicated search head and the associated search peers?

The rule of thumb is that one search is one CPU core, does that also apply to distributed search (per peer)? So if your search peers have 8 cores, 8 real-time queries will consume all the cores, effectively grinding the cluster to a halt. Is my understanding correct?

Is there a functional difference between Splunk versions (I am particularly interested in 5.0.3)?

Thanks

gregbujak
Path Finder

I realized that joins in the RT query caused the acceleration to mis-behave. In the face of fast moving data, each instance of the same RT query would eat its share of the CPU (on my machine it was 20% per request). However, while expensive, in 6.0 this has been fixed. So now complex RT queries (with joins) are accelerated and work as expected.

The reality is that with fast moving data, instances of different RT queries will quickly consume all the cpu resources and grind the system to a halt.

ShaneNewman
Motivator

Remember that searches run on the Indexers, the compiling of data and post processing of the data happens on the search heads.

0 Karma

gregbujak
Path Finder

HI Alek, thanks for the response. Limits.conf does specify the max number real-time searches, but it doesn't explain how it works. So what would happen if we allow 8 rt queries at the same time?

0 Karma

aholzer
Motivator

As far as my understanding goes (will have to owe you the documentation) a real-time search won't consume an entire core. What I mean by this is that multiple real-time searches will share cores and therefore increase how many concurrent real-time searches you can run.

Look in the limits.conf for the real time search limits and the concurrent limits. I think they give an explanation of how it works there.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...