Monitoring Splunk

How do I improve loading time of my dashboard with many scheduled searches running to populate it?

piebob
Splunk Employee
Splunk Employee

(asking this on behalf of another user)

I have one issue while creating dashboards in my app. To improve loading time in dashboard, I am using lookup created from scheduled searches which are running every 15 mins. Scheduled search is performing some calculations and using timechart command to place daily count in the lookup table. On dashboard time range picker is placed to select data from lookup for particular time range. Now there are multiple panels in dashboard. For each panel there are 5-6 scheduled searches and data is very large. Now it seems to be an issue, since scheduled searches keep on running and degrading the performance. Please suggest.

hcanivel
Explorer

What kind of saved searches are you performing? If they're mostly of using a time chart for a count, you're halfway there. Look into summary indexing if you haven't already: http://docs.splunk.com/Documentation/Splunk/6.0/Knowledge/Usesummaryindexing

See if any of the listed use cases apply to yours.

Definitely check out this if you haven't already:
http://docs.splunk.com/Documentation/Splunk/6.0/AdvancedDev/PostProcess

I'd consider a few other things (mainly taken from the above two references) if you wanted to cobble all these requirements together:

  • How related are these searches? Are they for the most part collecting from the same type of logs?
  • How expensive is each search? Is the purpose of this dashboard to summarize?
  • Do they need to be ad-hoc or real-time? Can you live with hourly updates if they truly are high volume searches/results?
  • How much data did you really want to consume out of this dashboard?
  • And finally/most importantly: is summary indexing ok?

If you can combine a lot of your requirements and using a base search, I think you should be able to achieve the rest. If you want to persist your scheduled searches, I'd recommend summary indexing and using adapted queries in this dashboard to just aggregate from there. Some overhead, but most efficient in the end potentially.

somesoni2
Revered Legend

can you give some example query you are using for panels?

Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...