How to increase performance of the complex searches in dashboard

[snehalk]

Hello Everyone,

I have created multiple dashboards but when i open the dashboard its taking too much of time to load and populated the result, here the main problem is we have to execute each query in dashboard for All time because of requirement and in that the query itself has some join functions to bring the required report.

So to make better performance i tried below option but still its not achieved my goal
1. Used post process searches for all the panal but here am enable to give end user download option
2. Used summary index but each schedule search is taking minimum 1 mins to completed the job so like this we have multiple saved
searches it might load to our search head and dashboard need to show real time data.

Could you please any one guide me how to increase the dashboard performance?

Thank you in Advance.

[tjago11]

You could use a combination of a Summary Index then supplement the data with what hasn't been summarized yet.

If you are running for "All Time" then you are likely doing some type of | stats avg(field1), count(field2) etc. You could keep this query, replace the stats with sistats and save the search as a summary index job that runs daily.

Then when you are in your dashboard you can query the summary index for any data that is greater than one day old and append that to today's data. Something like this:

index=summary source="YouSummaryData" earliest=-1w@d latest=-1d
| bucket _time span=1d 
| stats count as TransactionCount by _time 
| append 
    [ search index=YourOperationalData earliest=@d latest=now
    | bucket _time span=1d 
    | stats count as TransactionCount by _time
    ]

The trick here is to control the earliest and latest fields to rely mostly on the Summary data and only pull one day of your Operational data.

[koshyk]

if you put your query, it would be helpful

[hettervik]

Have you considered using scheduled searches to populate the dashboard panels?

https://answers.splunk.com/answers/418731/is-it-possible-to-schedule-a-dashboard-to-run-pane-1.html

[snehalk]

Hello Hettervi,
Thanks for link, is this take the reference of old schedule and combine the data with realtime and display the result?

[hettervik]

Well, what do you mean by real time? You could schedule the searches to run, say, every 5 minutes. If so the users could potentially see data that is 5 minutes old, worst case scenario. If you need the data to be seen in actual real time, then I guess you couldn't even use a summary index?

If you haven't already, you could also utilize a base search, and run sub-searches on that search, if possible. Though, this is maybe what you mean when you say that you use post process searches?

[snehalk]

Hello Hettervi,

I have done the approach of summary index but there is issue of real time as you mentioned. and also tried with base search technique but here we are not getting download data option to any of the panel in dashboard.

So is there any ways where i should get both the feature like realtime data in less time and download option too?

[hettervik]

I'm not quite sure what you mean with "download option", could you please elaborate?

What's important to have in mind with base searches is that it shouldn't return raw events. If you can do a stats command on the base search before sending the results to the sub-searches, or at least strip away as much data as possible, you maximise efficiency.

[snehalk]

Hello hettervi,

Its the "Export option" in each panel will see, ( which also has some other options too like Open in Search, Inspect)

Below is reference link
Please check this

[hettervik]

Aha, thanks. I didn't know export didn't work for post process searches.

[rjthibod]

In order to avoid some generalizations you have already touched, the community can best help you if you share details about the searches/reports you are running, the data types, etc.

The more details about the actual search queries, the more we can help.

[snehalk]

Hello Rjthibod,

Thank you for response, in the query am not using any knowledge object like eventtype, tag etc.
The query has some if else condition and one join condition, but main problem is am dealing with large volume of data (In that all time search).

Can you please guide me how i can refine or split this time range?

Thank you

[adonio]

there are many ways to make the dashboard work better, did you consider base searches? read here:
https://docs.splunk.com/Documentation/Splunk/6.5.2/Viz/Savedsearches
and here:
https://docs.splunk.com/Splexicon:Basesearch
@rjthibod makes very valid points, follow his lead. there are plenty of other ways as well, have you considered summary indexes? also what is the use case for "all time" data?
hope it helps

[rjthibod]

More details would be helpful. Can you share the actual query? If not, what fields are you using? How do the base searches between the join differ?

You will get the best help if you share the actual query.