Datamodel Acceleration: How to make DM acceleration searches fast?

[koshyk]

This query is for advanced tuning of Splunk Tiers so that the DM acceleration queries can run fast
http://docs.splunk.com/Documentation/ES/4.7.2/Install/Datamodels
We have already done
- Index specifics in DM , so it searches only specific Indexes
- Load balancing on Indexers to get fast data as fast as possible
- Reduced the retention as required and disabled unused DMs

Other suggestions in our Mind
1. to mount /opt/splunk/var/run in Search Head onto RAM (or SSD)
2. Customise the official TA's to remove unwanted fields for the customer. The effort vs return is NOT efficient here 😞
3. Override unwanted eventtypes/tags as per customer requirements

Any other suggestions from your side?

[woodcock]

Upgrade to the latest release that has no known issues for DMs.
Add more RAM to your Indexers.
Add more RAM to your Search Heads.
Add more Indexers.
Add more Search Heads.
Make sure pipelining is enabled (should be set to be equal to the number of CPU cores on that server).
Run the Health Checks form Monitoring Console and fix EVERYTHING (e.g. kill THP).
Make sure all of your searches are using summariesonly=true.
Hire a Consulting company to evaluate your environment and provide recommendations (there are many who do this, not just Splunk).

[koshyk]

Thanks woodcock.

Upgrade to the latest release that has no known issues for DMs. => Still to be done
Add more RAM to your Indexers. => Done. using 20% only
Add more RAM to your Search Heads. => Done using 25% only
Add more Indexers. => Have 48 of them
Add more Search Heads. => Why this one? The client has 7, but how it can improve. The searches are still going on and parallel, but slow.
Make sure pipelining is enabled (should be set to be equal to the number of CPU cores on that server).=> batch_search_max_pipeline is 2. Most of http://docs.splunk.com/Documentation/Splunk/6.6.3/Capacity/Parallelization is done
Run the Health Checks form Monitoring Console and fix EVERYTHING (e.g. kill THP). => Nothing much showing errors other than slowness in search results
Make sure all of your searches are using summariesonly=true. => The final searches are like that. But it is the "datamodel" acceleration searches which are the slow ones.

[woodcock]

I assumed this came up because searches are being skipped; if so, the surest way to fix that is more Search Heads. For all I knew, you only had 1 (not mentioned in your OP).