Splunk Search

Custom Index

mavilla
Explorer

I have one specific dashboard that I monitor with 7 tiles on it, there are times when the dashboard searches auto cancel and it is slow to load as well, once it is reloaded the results of the searched come back for a period of time then auto cancel again. Currently there is one main index for everything that's coming into Splunk, would setting up a separate index just for the searched in this specific dashboard fix some of these errors? All the searches in this dashboard need to be real time.

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

First, it's rarely a good idea to put all data into a single index.
Second, putting the data into separate indexes probably won't help as the limiting factor is number of search slots (CPUs) available at the moment.
Third, a dashboard does NOT need to be real-time. Real-time searches should be used when the response will be real-time. For people watching a dashboard, a 1-minute or 5-minute interval is sufficient. By using real-time searches, your dashboard is tying up 7 CPUs on the search head and all indexers.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

First, it's rarely a good idea to put all data into a single index.
Second, putting the data into separate indexes probably won't help as the limiting factor is number of search slots (CPUs) available at the moment.
Third, a dashboard does NOT need to be real-time. Real-time searches should be used when the response will be real-time. For people watching a dashboard, a 1-minute or 5-minute interval is sufficient. By using real-time searches, your dashboard is tying up 7 CPUs on the search head and all indexers.

---
If this reply helps you, Karma would be appreciated.

mavilla
Explorer

Thank you for your answer Rich - just so I understand this correctly, you're saying that since I have 7 tiles on this dashboard and each of them is searching in real time, each tile is taking up one CPU to perform the searches? For context, our Splunk server is running 16 CPU cores (virtual), I see your point about the entire dashboard being in real time.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Yes. Seven real-time tiles/searches take up 7 CPUs, not only on the search head where the dashboard runs, but also on all indexers.

---
If this reply helps you, Karma would be appreciated.
0 Karma

mavilla
Explorer

Our search head and indexer are hosted on the same server, we've already identified this as being an issue but didn't know it was this much of an issue. Thanks again

0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...