Splunk Search

Splunk.Module.Search Context failed to dispatch job for search XXXX

guilhem
Contributor

Hello,

I have a dashboard where I do several search/ display and randomly I have the error given in the title so I would like to know the possible causes of this error.

As there is no more information given in the JS console (and no other errors, except an assert on TimeRangePicker: Assertion Failed - we have no selected range. If this occurs with the calendar-pickers its a possible race condition.) it is pretty hard to tell what's happening. Other than it is pretty random, except on my computer where all the search run just fine.

thanks!

EDIT:

The dashboard I have include 10 searches that are run when the page loads. When I keep only the first three, no error, but when I had the 4th and 5th, the error occurs for both of them.

I strongly suspect it is a server response issue (maybe laggy or something), but it may not. If anyone have an idea, it would be appreciated. Maybe it is possible to delay the search request so that the server doesn't get overhelmed.

And as I have stated in my comment, it is strongly related to the type of user, as normal user have the error, but admin do not (haven't tested for power user).

Tags (1)
0 Karma
1 Solution

sideview
SplunkTrust
SplunkTrust

If the users getting the error are non-admin users and you're not getting it as an admin, that is a strong clue that the issue is the user limit on the number of concurrent searches.

For admins the limit is quite high, but for regular users it's somewhere much lower. Possibly 3.

Ideally, searches in a dashboard should either be
a) loaded from a scheduled saved search
b) a shared real-time search.
c) run ad-hoc, but completing very very fast, ie running against a summary index.

for a and b, this means that the user loading the dispatch doesn't spend any dispatched searches at all - the searches are already dispached and access is shared.

for c, the searches dispatch and complete so quickly that even with only being able to run 3 at a time, the lowly user-level users can still get all 10 completed before the failover messaging tells them that there's something wrong and there searches can't be dispatched.

The obvious answer might be to raise the number of concurrent searches for all users, but this is usually not a great idea because you trade per-user limits which are irritating, for system-wide limits and if those system-wide limits are also circumvented, system-wide performance problems around running too many concurrent searches globally.

View solution in original post

sideview
SplunkTrust
SplunkTrust

If the users getting the error are non-admin users and you're not getting it as an admin, that is a strong clue that the issue is the user limit on the number of concurrent searches.

For admins the limit is quite high, but for regular users it's somewhere much lower. Possibly 3.

Ideally, searches in a dashboard should either be
a) loaded from a scheduled saved search
b) a shared real-time search.
c) run ad-hoc, but completing very very fast, ie running against a summary index.

for a and b, this means that the user loading the dispatch doesn't spend any dispatched searches at all - the searches are already dispached and access is shared.

for c, the searches dispatch and complete so quickly that even with only being able to run 3 at a time, the lowly user-level users can still get all 10 completed before the failover messaging tells them that there's something wrong and there searches can't be dispatched.

The obvious answer might be to raise the number of concurrent searches for all users, but this is usually not a great idea because you trade per-user limits which are irritating, for system-wide limits and if those system-wide limits are also circumvented, system-wide performance problems around running too many concurrent searches globally.

guilhem
Contributor

And for admin it is 50. raising the cap to 10 concurrent search per user did the trick. Thanks!

0 Karma

guilhem
Contributor

I arrived at the same conclusion this morning. The elegant way to solve this problem would be to use only saved search, but as I am using sideview and a lot of $foo$ token replacement it is not really possible (or I am not aware of a way to do it).

EDIT: It looks like the maximum number of concurrent search for non-admin is 3:
http://splunk-base.splunk.com/answers/40789/concurrent-searches

0 Karma

guilhem
Contributor

OK, it is related to the fact that I am admin and user that have the errors are not.

Still investigating.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...