Why are my post process searches running slower than individual searches for panels in my dashboard?

[jagadeeshm]

Someone in my team created a dashboard with 8 panels. Each panel uses individual searches, for example:

Panel -1

index=xyz sourcetype=log_mover result=logmover_success | stats .....

Panel -2

index=xyz sourcetype=log_mover result=logmover_failure | stats .....

Panel - 3

index=xyz sourcetype=log_mover result=transfer_success | stats ......

and so on...and finally, Panel - 8 does an overall chart:

index=xyz sourcetype=log_mover result=* | stats....

All the Panels are built around the "result" field which has different values. But Panel - 8 covers all the different values of result (that is why result="*" in Panel - 😎

Looking at the queries I realized, this is definitely a candidate for post-processing using a base search. I redefined the panels, by creating a base search (which is same as Panel - 😎 and then subsequently updated all the Panels to use the base search for charting.

Base Search

<search id="baseSearch">
      <query>index=xyz sourcetype=log_mover result=*</query>
      <earliest>-24h@h</earliest>
      <latest>now</latest>
</search>

And each of my Panels use post-processing, for example , Panel - 1

<search id="firstPostProcess" base="baseSearch">
          <query>search result=logmover_success | stats .... </query>        
</search>

And likewise, I have set up all other panels. But I noticed my dashboard with post processing searches are running extremely slower than individual search. What am I doing wrong?
If I were to measure the performance with a timer, they are at least 3 times slower than individual searches. Thoughts?

[somesoni2]

The best way to utilize the post-process feature is to do at least some level of aggregation on the base search. The post process search has limitation of number of results that it returns and an aggregation will mitigate that. By seeing the you base search you're not doing any aggregation there. If you can share your actual searches (for all panels/variations available), we can suggest an aggregation which will work in your case and should improve the performance.

[bmacias84]

Post process search are inherently slower than individual. Typically I point people to post process searches when users or dashboards is maxing out system or user level concurrent searches. In short Post process search will always be slower, except for a very some set of cases.

Post process searches don't count against a user concurrency limit.

[jagadeeshm]

May I ask your source?

[bmacias84]

Slowness/performance of Post process search isn't well documented, but is a limitation/pitfall of using the technique. It commonly known among Splunk veterans. Not to say you cant optimize your base search to increase efficiency.

Here is some information on Post Processes. http://docs.splunk.com/Documentation/Splunk/6.5.0/Viz/Savedsearches#Post-process_searches.

Post from sideview utils (former Splunk Employee) third paragraph. https://answers.splunk.com/answers/85486/sideview-utils-postprocess-slow-speed-searchhead-or-indexer...

[Masa]

I agree with both somesoni2 and bmacias84 .

PostProcess sometimes helps for similar searches. But, in many cases, it becomes slower.

Even when aggregation search is used at base-search, it is still possible to make it slower. It depends on data and searches for a dashboards, either PostProcess got benefit or not.

In my personal experience, some dashboard performance was improved while the other didn't.