Splunk Search

Sub-search doesn't work in some apps/dashboards but works in others and as a separate search

michael_sleep
Communicator

Been wrestling with this issue for a while now... I have a search like the below (sensitive information redacted). This works when using it in a regular search. When adding it as a dashboard panel it doesn't work. The weird thing is, it works in some dashboards/apps but not others:

index=xxxx_logs container_name="organization-api" namespace=pvcs "update" UserController
| spath input=log
| search action=update 
| eval corrId=spath(log,"request_id")
| eval params=spath(log,"params")
| rex field=params "(?P<parameters>[^\"]+\"\:\"[^\"]+)" max_match=0
| rex field=parameters mode=sed "s/\"\:\"/: /g"
| eval userID=spath(params,"id")
| eval username=spath(params,"login_name")
| join corrId 
   [search index=xxxx_logs container_name="lbo" audit
    | spath input=log
    | eval corrId=spath(log,"corrId")
    | eval byUser=spath(log,"byUserId")]
| table _time,controller,action,userID, username,parameters,request_id,byUser
| sort _time

The only error that comes up in the job inspector is:
"No matching fields exist"

This says to me that the search can't or doesn't run spath or for one reason or another can't extract the fields necessary to do the sub-search. The weird thing is, I can copy/paste this into another dashboard or run it separately as a search and it works fine. I've gone through the app and the dashboard source but I don't see anything that would be different between them configuration wise.

I've tried including | fields=* to try and force verbose mode. I've also tried setting permissions as high as they'll go on the app/dashboard hoping this would fix it but nada... thoughts?

0 Karma
1 Solution

michael_sleep
Communicator

Nevermind, I figured this out coincidentally shortly after posting... it appears the problem is with using ADFS as SSO into Splunk. The created SIDs for search are too large and cause problems when attempting to create sub-searches. I resolved the issue by making a generic "admin" account owner of the dashboard panels, instead of my ADFS account.

View solution in original post

0 Karma

michael_sleep
Communicator

Nevermind, I figured this out coincidentally shortly after posting... it appears the problem is with using ADFS as SSO into Splunk. The created SIDs for search are too large and cause problems when attempting to create sub-searches. I resolved the issue by making a generic "admin" account owner of the dashboard panels, instead of my ADFS account.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...