Dashboards & Visualizations

Why is my dashboard panel returning loadjob error "accessing https:///saved/searches/External API Usage/?output_mode=json, statusCode=502, description=Not Found"

tkwaller
Builder

I have a dashboard (within an app) that uses the loadjob command that usually works, but now returns:

Error in 'SearchOperator:loadjob': error accessing https:///saved/searches/External API Usage/?output_mode=json, statusCode=502, description=Not Found

Any ideas what the issue could be?

I have the saved search saved in savedsearches.conf and its scheduled and running successfully

1 Solution

sansay
Contributor

As we have a search head cluster and I wanted to use loadjob, I ran a bunch of experiments. Here are the results:
Version 6.3.3.0.1 loadjob command will only work if the 3 following requirements are respected:
1. saved search name must not include spaces
2. saved search must be added to the search app (I had to go around through many test to make sure of this issue, but if your searches are in a custom app you will get this error message: "Error in 'SearchOperator:loadjob': Cannot find artifacts for savedsearch_ident 'user:custom_app:saved_search'.
3. saved search permission must be public if you want others to see the results retrieved by this command

Also, keep in mind that the saved search must be scheduled and the loadjob call should happen within a time range where the job results haven't been automatically deleted in order for you to see any previously collected results.

View solution in original post

sansay
Contributor

As we have a search head cluster and I wanted to use loadjob, I ran a bunch of experiments. Here are the results:
Version 6.3.3.0.1 loadjob command will only work if the 3 following requirements are respected:
1. saved search name must not include spaces
2. saved search must be added to the search app (I had to go around through many test to make sure of this issue, but if your searches are in a custom app you will get this error message: "Error in 'SearchOperator:loadjob': Cannot find artifacts for savedsearch_ident 'user:custom_app:saved_search'.
3. saved search permission must be public if you want others to see the results retrieved by this command

Also, keep in mind that the saved search must be scheduled and the loadjob call should happen within a time range where the job results haven't been automatically deleted in order for you to see any previously collected results.

stanwin
Contributor

The search context issue is confirmed!

Loadjob is not able to pickup any saves searches outside of search app!!

Below is the issue:

Error in 'SearchOperator:loadjob': Cannot find artifacts for savedsearch_ident 'admin:JLPOperations:birdseye_rtr_base'.
The search job has failed due to an error. You may be able view the job in the Job Inspector.

Splunk 6.3.4 (build cae2458f4aef)

0 Karma

tkwaller
Builder

Yes, absolutely. Haven't been able to get back to update this but this is what I found as well. All 3 points MUST be met or it will fail. Good testing!

0 Karma

tkwaller
Builder

So for that, I give you the answer credit!

0 Karma

sansay
Contributor

Thank you!
Glad to see that I am not the only one to reach this conclusion.

0 Karma

czeller
Engager

I've encountered this as well, and I believe it's related to search head clustering as it only started happening after we upgraded to 6.3. Using |savedsearch was not an option for me, because it negates the primary effect of using |loadjob, namely, that I can schedule a long-running search to execute in the background and then pick up the results when my report or dashboard needs it.

I am able to use this as a workaround:

|loadjob [rest /services/saved/searches/mysearchname/history splunk_server=local |where isDone==1 |eventstats max(ttl) as maxttl |where ttl==maxttl |return $title]

Replace mysearchname with the name of your saved search.

anselmhartl
Engager

The path "/services/saved/searches/xx_xxx_ENV-All_Report_allT_ENV/history" did not work.
=> Permission denied. Cannot access artifacts of job_id ''.

The path "/servicesNS/nobody/xx_rp_xxx_portale/saved/searches/xx_xxx_ENV-All_Report_allT_ENV/history" worked.

|loadjob [rest /servicesNS/nobody/xx_rp_xxx_portale/saved/searches/xx_xxx_ENV-All_Report_allT_ENV/history splunk_server=local |where isDone==1 |eventstats max(ttl) as maxttl |where ttl==maxttl |return $title]

0 Karma

stanwin
Contributor

Splunk 6.3.4

This workaround works , Excellent czeller!

However you need to set global read permission for the savedsearch for rest to access it, atleast in my case .

Below errors are seen but dont affect the resultset or graph..

8 errors occurred while the search was
executing. Therefore, search results
might be incomplete. Hide errors.
[subsearch]: [SPLUNK_SERVER01] Failed
to fetch REST endpoint
uri=https://127.0.0.1:8089/services/saved/searches/my_saved_search/history?count=0
from server=https://127.0.0.1:8089
[subsearch]: [SPLUNK_SERVER02] Failed
to fetch REST endpoint
uri=https://127.0.0.1:8089/services/saved/searches/my_saved_search/history?count=0
from server=https://127.0.0.1:8089
[subsearch]: [SPLUNK_SERVER03] Failed
to fetch REST endpoint
uri=https://127.0.0.1:8089/services/saved/searches/my_saved_search/history?count=0
from server=https://127.0.0.1:8089

0 Karma

tkwaller
Builder

OK so here's the deal:
Apparently in 6.3.2 "| loadjob" for whatever reason doesn't work in the clustered search head environment. I tested this by changing the search string from:

| loadjob savedsearch="tkwaller:api-gateway-tools:External API Usage" | where appName="XXXX" | stats count by apiTarget | rename count AS NumberOfCalls | sort -NumberOfCalls

to

| savedsearch "External API Usage" | where appName="XXXX" | stats count by apiTarget | rename count AS NumberOfCalls | sort -NumberOfCalls

Everything works good

sansay
Contributor

Actually this will just rerun your saved search. The whole idea of loadjob is to retrieve results from the last run of the saved search.

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

You may need to URL-encode your url, because it has spaces in it. Take a look here for documentation on the urlencode eval function.

0 Karma

tkwaller
Builder

It is useful but there is no urlencode there, just urldecode

0 Karma

tkwaller
Builder

So I created a new scheduled search(with no spaces) the search is:
| loadjob savedsearch="tkwaller:api-gateway-tools:External_API_Usage"

it now returns:
Error in 'SearchOperator:loadjob': Cannot find artifacts for savedsearch_ident 'tkwaller:api-gateway-tools:External_API_Usage'.

If I omit the "| loadjob" command and just use "savedsearch="tkwaller:api-gateway-tools:External_API_Usage"" I get back:
Savedsearch 'tkwaller:api-gateway-tools:External_API_Usage' does not exist or is disabled.

but it does exist and is readable and writable to Everyone

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...