Dashboards & Visualizations

Why do saved searches take too much time in dashboards?

Javo222
Path Finder

I have several dashboards with 8 to 15 graphs/saved searches each and when I try to display them I get "The maximum number of historical concurrent system-wide searches has been reached. current=6 maximum=6 SearchId=admin_admin_search..." I understand why and I tried increasing base_max_searches = 2 but my CPU maxes out at 100% and my PC freezes.

What I would like to do is to run maybe 6 concurrent searches and when these are do, run the next batch of 6.

Other solution I thought of could be to used cache data/searches but I wonder about the following points:
- How can I cache a saved search so it's not executed each time I open the dashboard?
- I stop my Splunk service whenever I'm done with looking at the dashboard. Some weeks later, I will need to look at them again and will start Splunk. Can the saved searches be preserved?
- My saved searches have a cron schedule in savedsearches.conf, similar to cron_schedule = 30 20 1-31 * * with different time of the day.

I barely need to refresh my data, so I don't mind having to use the data cached even if it's 1 month old...

Which parameter can allow me to implement one of these solutions? Or can you think of a better way?

I got this Splunk setup from someone else so I'm not exactly sure about the entire configuration.

I'm using Splunk Enterprise 6.3.0

0 Karma

woodcock
Esteemed Legend

Are you running any realtime searches? If so, that is your entire problem. DO NOT use realtime, especially in a dashboard.

0 Karma

cmeerbeek
Path Finder

This related post has a good answer which points you in the right direction:
https://answers.splunk.com/answers/233092/how-to-cache-app-dashboard-to-avoid-search-query-e.html

You need to use scheduled searches and call those reports. Splunk will first try to get the cached results and will not execute the search.

If you search for "Splunk dashboard saved searches" or "Splunk dashboard cache" you will find other topics about this matter.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...