Splunk Search

Real Time Search Issues

shanecifaldi
Loves-to-Learn Everything

We're running into an issue where are RT searches are being delayed due to the amount of concurrent searches being ran.

priority=default, status=delayed, reason="The maximum number of concurrent running jobs for this real-time scheduled search on this instance has been reached", concurrency_category="real-time_scheduled", concurrency_context="saved-search_instance-wide", concurrency_limit=1, scheduled_time=1556040360, window_time=0

I have double checked everything in my limits.conf that could stop these searches.

base_max_searches = 7
max_rt_search_multiplier = 4
max_searches_per_cpu = 4 (4 cpu 18gb ram)

max_searches_perc = 100
auto_summary_perc = 100

We're on version 7.2.4.2 - cron alerts fire with no issues but RT do not. I know several people will say dont use RT alerts - not interested in your opinion in that regards - just whats holing up my own RT searches.

As you can see below - we dont have that many searches running.

alt text

0 Karma

woodcock
Esteemed Legend

You cannot have more RT searches than CPU cores, but you can cheat and use fake RT:
On Search Heads in limits.conf:

[realtime]
#https://docs.splunk.com/Documentation/Splunk/latest/Search/Aboutrealtimesearches#Indexed_real-time_search
indexed_realtime_use_by_default = true
0 Karma

woodcock
Esteemed Legend

I suggest that you give up on real-time searches for exactly this reason, among many other good ones: https://answers.splunk.com/answers/734767/why-are-realtime-searches-disliked-in-the-splunk-w.html

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...