Splunk Search

Why and when is Splunk differing between canceling and queuing searches?

HeinzWaescher
Motivator

Hi,

I'm wondering why (and when) there is a different handling when a lot of searches are running at the same time

  • List item waiting for your queued job

vs.

  • List item "Search not executed: The maximum number of concurrent historical searches on this instance has been reached."

For a dashboard user option 1 is much better, because he will see the needed results in the end. When option 2 is used, the dashboard will not complete

gcusello
SplunkTrust
SplunkTrust

Hi HeinzWaescher,
usually searches are queued when they reach the maximum number of concurrent historical searches.

The best way to solve the problem is possibly upgrade your hardware (CPUs) and modifying limits.conf.

At the same time it could be a good idea (to use everytime) to try to optimize your dashboard using post processing search (see https://docs.splunk.com/Documentation/Splunk/latest/Viz/Savedsearches ), you can find an example in Splunk 6.x Dashboard Examples app.

Bye.
Giuseppe

HeinzWaescher
Motivator

Hi cusello,

thanks for your reply. We know about the option to modify hardware & configs. But the issue would not be such a problem, when all searches would be queued. Then it would simply take longer to load the dashboard.
But we often see these canceled search, which finally means that the dashboard won't finalze. But I haven't found a pattern when it happens.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi HeinzWaescher,
Maybe some of your searches are timeouted.
if you open a search in the Search dashboard can you run it?
after result, see job inspector and see if meybe there is a timeout.
Bye.
Giuseppe

0 Karma

HeinzWaescher
Motivator

I can't click the "open in search" button for these searches. The job Inspector says "unknown sid"

0 Karma

gcusello
SplunkTrust
SplunkTrust

open in search before they are timeouted.
Bye.
Giuseppe

0 Karma

HeinzWaescher
Motivator

The error appears instantly when the dashboard is opened and Splunk tries to run to many searches. So there is no time until a timeout so that I could crosscheck it.
(Or I just don't understand what you mean :))

0 Karma

gcusello
SplunkTrust
SplunkTrust

take the search from your dashboard source and execute it in search (manually inserting eventual parameters) , to see if the problem is a timeout.
Bye.
Giuseppe

0 Karma

HeinzWaescher
Motivator

ah okay 🙂 that works fine, starting instantly and finalizing fast.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Try to rebuild your dashboard search by search and then putting inputs, maybe there's an error in parameters passing.
Bye.
Giuseppe

0 Karma

HeinzWaescher
Motivator

I think I got it.
I created two dashboards. The first included saved non-scheduled searches, the second includes the same searches as inline searches.

I opened both dashboards:
The first dashboard started calculating results for max limit of concurrent searches, the rest was cancelled.
In the second dashboard all inline searches were queued and finalized step by step, nothing was cancelled. So it depends how the search in implemented in the dashboard.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...