How to find and stop real time searches running on indexers?

[kteng2024]

Hi there,

I am seeing some real time searches running on indexers. Can I please know how real time searches are running on indexers as they should be running on search head and also is there any query to find these type of searches running on indexers? Moreover, how to stop these kind of searches as they are consuming to much of CPU?

[s2_splunk]

Any search started on a SH will also start a search on each indexer. If you initiate a realtime search on a SH, a process will kick of on each indexer to satisfy that search request and only end if the search is stopped on the SH. That's how distributed search works for realtime searches.

It's generally good practice for a production environment to disable users' ability to run realtime searches due to the resource impact they have. Maybe restrict it to just power users that are trained properly and understand the impact, and/or consider indexed realtime searches, which are scheduled searches with a very small interval.

There are very, very few use cases that actually require realtime searches. For time-critical alerts a 2-5 minute interval on a scheduled search is almost always sufficient.

This post should help answer your question about how to find them.

[kteng2024]

Thank you for the reply . It is really helpful. But I also see scheduled searches running on indexers.So,it means all the searches started by search head will be dispatched to indexers .But can I please know even the scheduled search is stopped on SH, how come searches are still running on the indexers ?

[DalJeanis]

Think of it this way - each indexer is like a library with its own librarian. Whenever you start a search, you give a request to the head librarian, and she writes instructions for all the other librarians to check their libraries for the books you have asked for, and instructions on what to send back (the whole book, or just a couple of pages).

When you cancel a search at the search head, the head librarian has to write an instruction to cancel the searches at each library, then the individual librarians need to get that order. If they happen to be back in the back shelves and storerooms getting information for your search, or for another search, there might be a delay before they know they are supposed to stop your search.

II recall correctly, there is a ten minute default duration for searches, so ten minutes after the search head is brought down, all of the subsidiary searches should have completed... although I'm fuzzy on the exact process order so I'll defer to wiser heads.