Hello, I have updated a Splunk Cluster from V7.3.8 to V8.1.2 following the documentation provided by Splunk and since the update we have an issue with the scheduled searches Schedule searches are running normally after a Searchhead Cluster restart but after some time they skipping on the Capitan and and they do not run at all on the other nodes In the screenshots above Scheduled Searches where running until 8 AM CET and then all are skipped on the Captain and the other Search Heads did not process any Scheduled Searches. I found a workaround to move the Captain to another SearchHead and then Schedules Searches will run again. As seen in the example above The cluster is composed of 3 Indexers, 3 SearchHeads and 1 Master node I have increased the Relative concurrency limit for scheduled searches to 70% and Relative concurrency limit for scheduled searches to the same 70% Also adapted the limits.conf to # The base number of concurrent searches. base_max_searches = 60 # Max real-time searches = max_rt_search_multiplier x max historical searches. # max_rt_search_multiplier = 1 # The maximum number of concurrent searches per CPU. max_searches_per_cpu = 10 max_searches_perc = 60 But nothing helps A sure way to reproduce this on the system is to stop one of the SearchHeads and then start it. Aprox 10 Minutes after the SearchHead starts all scheduled searches will be skipped on the Capitan In the Logs there is only one type of "Error" (actually info message) : _ACCELERATE_AF2AEFDE-8E13-4DCA-90CB-C21D356D9A60_iqpress_nobody_e0c3b6f1a41c2518_ACCELERATE_ The maximum number of concurrent historical scheduled searches on this cluster has been reached (220) Thank you very much in advance 🙂
... View more