Dashboards & Visualizations

Why does this search fail when the dashboard panel loads, but runs successfully when executed manually.

abeeber_2
Path Finder

Hi everyone,

I have a power user search that I am having problems with.

When the dashboard loads the search in one of the panels fails, I get a subsearch error from each indexers in my deployment.

Yet if I open the search in a separate window (Open in Search) it works.

Does anyone know what would cause this?

Thanks,

SPL Below>>>
index=testsearch sourcetype=transaction (application="SharePoint Static" OR application="SharePointStatic" OR application="SharePointUpload" OR application="SPTeamSiteMySite*") transaction="*UploadTestFile3MB" transaction_status="Success"

|eval secs=round(duration/1000)
|lookup local=1 testsearch_network_mapping_file.csv hostname OUTPUT longitude latitude UserCount DisplayName
|search (UserCount > 0 AND UserCount < 100000000) DisplayName != "Frank - Campus (" |rangemap field=secs Good=0-3 Fair=3-4 Bad=4-2000

|stats count AS TransTot max(UserCount) AS Users max(longitude) AS long max(latitude) AS lat BY DisplayName range
|join DisplayName [search index=testsearch sourcetype=transaction (application="SharePoint Static" OR applicaton="SharePointStatic" OR application="SharePointUpload" OR application="SPTeamSiteMySite
") transaction="*UploadTestFile3MB"
|lookup testsearch_network_mapping_file.csv hostname OUTPUT DisplayName
|stats count AS SiteTotal by DisplayName]
|eval ChartValue=TransTot*Users/SiteTotal
|table *
|geostats latfield=lat longfield=long sum(ChartValue) by range
|eval s=Good+Bad+Fair
|eval Good%=round((Good / s) * 100,2)
|eval Bad%= round((Bad / s ) *100,2)
|eval Fair%=round((Fair / s) * 100,2)
| fields - s

0 Karma

abeeber_2
Path Finder

Here is my fix/RCA.

Turns out the root cause was using Windows 2008R2 for the index tier.

Using SSO AD accounts that have FQDN meant the hashed value of the search sid exceeded the character limit of the server. This was identified by using the | history command to see the difference.

Ultimately, we're migrating the index tier to RHEL and as a workaround, I'm creating shared local accounts to facilitate access.

0 Karma

abeeber_2
Path Finder

More intel.

Turns out there is a bug/fix in Splunk 6.4.5 where they shortened a temp file from 30 characters to 16.

We installed 6.4.9 on the index tier and the problem went away.

0 Karma

hettervik
Builder

The search isn't by chance derived from a base search? If so, you might have to define which fields are going to be handed down by the base search to the sub-searches in the dashboard.

0 Karma

abeeber_2
Path Finder

@hettervi,

good question. I am working through the syntax to figure that out. I'm thinking the join might be part of the problem.

0 Karma

MuS
SplunkTrust
SplunkTrust

So what is the error message you get from the indexers? Can you please check the job inspector and post the error as well, please ?

0 Karma

abeeber_2
Path Finder

MuS,

Here are the indexer related errors..

The indexers are windows 2008R2 (to be migrated to RHEL very soon).

error : [subsearch]: [PRODASRV235] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODASRV236] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODASRV274] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODASRV354] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODASRV411] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODASRV423] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODASRV630] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV236] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV237] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV238] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV239] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV321] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV374] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV380] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV381] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [PRODBSRV630] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [KULAPP235] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [KULAPP236] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [LHDAPP235] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
error : [subsearch]: [LHDAPP236] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
(SID: YXJiZWViZUBuYS54b20uY29t_YXJiZWViZUBuYS54b20uY29t_MTAwX1hPTV9FMkVfc2hhcmVwb2ludA_search1_1511303688.4262_588B113C-A2D3-42E1-ACE2-8E5D47E3C0DE) search.log

0 Karma

abeeber_2
Path Finder

Would the length of the sid be a problem? I noticed that when I run the search manually, the sid is much shorter.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...