Splunk Search

Why are search results different when running a search in the Search app versus a dashboard panel?

splunked38
Communicator

Hi,

I have a search as a dashboard panel.

When I execute the search on the dashboard, the result is incorrect.

What's interesting is:

  • If I refresh the panel, the result is still incorrect
  • when I 'open the search' from the dashboard panel, it's still incorrect
  • When hit the search button after the 'open the search' the result is correct.

Unfortunately, I can't post up the search.

Notes:

  • At no stage was the search changed
  • The time selector was not changed
  • The search was working in Splunk <6.3.0
  • If I dissect the search, the individual components return the expected result

I'd be interested as to what I can do to check to see where the problem could be.

Tags (3)

splunked38
Communicator

I've 'narrowed down' the problem:
the search in the dashboard panel looks like this:

search xxx | stats count(Name) as search1
| eval search2=[search yyy | stats count(Name) as search2| rename search2 as query]
| table search1, search2

search1 is ok, search2 produces the wrong result but if I did this in the dashboard panel:

search yyy  | stats count(Name) as search2 | table search2

The correct result comes up.

Just to re-iterate:
if I go to the dashboard panel, hit 'open in search', the panel search will appear, I hit 'search', and the correct result appears. Because of this, I'm leaning away from the fact that the syntax is the issue. Something about the way the dash executes the search provides incorrect results.

Before you ask, the reason why I need the two searches in the one dash is because I'm trying to get a percentage from the two searches:

search xxx | stats count(Name) as search1
| eval search2=[search yyy | stats count(Name) as search2| rename search2 as query]
| eval percent=round((search2/search1)*100,2).%

0 Karma

Sebastian2
Path Finder

I'm very new to Splunk (so my answer may be stupid) - but the first thing coming to my mind is the app context. Is the normal search maybe performed in another app? Do you have anything configured, like transforms combined with auto lookups which affect your search in any way?

0 Karma

splunked38
Communicator

Nope, dash is in search.

If it was app context, why would the second attempt in search be different, I would expect it to be the same

0 Karma

sundareshr
Legend

Are you using sub-search? If the sub-search reaches the limit, it will return 0 results, thereby affecting the results of the main search.

sundareshr
Legend

If you look at the job inspector after you run the search in the main search window (after it returns results successfully), is there anything re: results being truncated? If there is, that is the reason dashboard results are inaccurate.

Something else to look for, when you hit "open in search" in dashboard panel, BEFORE you hit the search button, do you see the results? Go look at the "normalizedSearch" in the job inspector window. Does that look correct?

One more place to check - Run the search in dashboard and search window in quick successing. Go to "Activity->Jobs. Compare the Events count from all 4 activities listed (2 for main search, 2 for subsearch). Are they the same?

In my experience, if the result for a sub-search is truncated, the results in dashboard is not the same as what you see in the main search window. Don't know why, just seen this happen.

The other thing to verify is permissions and app context... do those look correct?

lmonahan
Path Finder

Super helpful checklist, thanks!  Also will include a link to this answer as base queries with non-transforming results have boundary limits. I think if you're getting inconsistent results its because you're over a limit.

https://community.splunk.com/t5/Splunk-Search/Post-processing-gives-incorrect-results/m-p/522520

 

0 Karma

splunked38
Communicator

I compared the normalised searches between the two and they are both the same.

Both the 'good' and the 'bad' search are truncated.

Perms and app context are both accessible within the application.

0 Karma

splunked38
Communicator

ok, that's understandable

however, the search string returns the correct result when I manually copy and paste it into search.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...