Deployment Architecture

What do I do if I have too many saved searches with status="continued" in my search head cluster?

ishaanshekhar
Communicator

I have a search head cluster with quite a few saved searches that run every 5 mins.

Sometimes, the status of a few saved searches become "continued". I understand that system will come back to that later, but quite often the queue gets longer and longer and continues until I restart the SHC.

I would like to know what should I do in this scenario? options:

1) Add another SH in the cluster
2) bump resource values in limits.conf
3) increase the frequency of savedsearches to distribute the load (last option)
4) or anything else?

Thanks
Ishaan

0 Karma

teunlaan
Contributor

What you can do, depends on the reason why there are not running immediately.
So before you try to fix anything find the error.

1) look at the duration of your searches. If you run the every 5 minutes, they should be finished within the 5 minutes.
2) If searches take a long time to finish, take a look at the search. Are the build in the right way (specify host, source type, ect.)
3) Are the searches “continued” because you don’t have any CPU-cores left on you searchheads OR on you indexers.

- Maybe more Indexers will help (single indexer has to search a smaller amount of data).

- Extra Searchhead is only usefull, if your Indexers can handle it

0 Karma

mreynov_splunk
Splunk Employee
Splunk Employee

The searches are actually performed on the indexers, so I would look at that first. Here is a bit of reading to get you started: http://docs.splunk.com/Documentation/Splunk/6.2.0/Deploy/Datapipeline

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...