Dashboard with 2 searches hangs and only 1 displays

[orweinberger]

I have a dashboard with 2 saved searches loaded to it, when I try to load the dashboard it takes around 1-2 minutes for the graphs to render (only 1 is able to render each time - I have no idea why) and the CPU/MEM on the splunk machine spikes.

Here are the two saved searches:

This should show the avg of mem usage for the process 'splunkd' on two hosts

index="os" sourcetype="ps" host=stage.site OR host=testbuild.site | multikv fields pctMEM, COMMAND | where COMMAND="splunkd" | append [stats sum(pctMEM) as pctMEM by _time,COMMAND | eval host=split("stage.site;testbuild.site",";")
| mvexpand host ] | timechart avg(pctMEM) by host | eval count=count-1

This should show the avg of cpu usage for the process 'splunkd' on two hosts

index="os" sourcetype="ps" host=stage.site OR host=testbuild.site | multikv fields pctCPU, COMMAND | where COMMAND="splunkd" | append [stats sum(pctCPU) as pctCPU by _time,COMMAND | eval host=split("stage.site;testbuild.site",";")
| mvexpand host ] | timechart avg(pctCPU) by host | eval count=count-1

The time range for these two searches is defined to -24h@h.

I have another dashboard with the exact same saved searches but with a different time range of -1h@h and that dashboard loads correctly without spiking the mem/cpu usage and it does not hang.

This is how I load the 2 saved searches in my dashboard:

<module name="HiddenSavedSearch" layoutPanel="panel_row2_col1" group="splunkd CPU Usage last 24h" autoRun="True">
    <param name="savedSearch">splunkd CPU Usage last 24h</param>
    <param name="groupLabel">splunkd CPU Usage last hour (24h)</param>
    <module name="HiddenChartFormatter">
      <param name="charting.chart.stackMode">default</param>
      <param name="charting.secondaryAxisTitle.text">% CPU</param>
      <param name="charting.legend.placement">left</param>
      <param name="charting.chart">line</param>
      <param name="charting.chart.nullValueMode">connect</param>
      <module name="JSChart">
        <param name="width">100%</param>
        <param name="height">280px</param>
        <module name="ConvertToDrilldownSearch">
          <module name="ViewRedirector">
            <param name="viewTarget">flashtimeline</param>
          </module>
        </module>
      </module>
      <module name="ViewRedirectorLink">
        <param name="viewTarget">flashtimeline</param>
      </module>
    </module>
  </module>
    <module name="HiddenSavedSearch" layoutPanel="panel_row2_col2" group="splunkd MEM Usage last 24h" autoRun="True">
    <param name="savedSearch">splunkd MEM Usage last 24h</param>
    <param name="groupLabel">splunkd MEM Usage last 24h</param>
    <module name="HiddenChartFormatter">
      <param name="charting.chart.stackMode">default</param>
      <param name="charting.secondaryAxisTitle.text">MEM %</param>
      <param name="charting.legend.placement">left</param>
      <param name="charting.chart">line</param>
      <param name="charting.chart.nullValueMode">connect</param>
      <module name="JSChart">
        <param name="width">100%</param>
        <param name="height">280px</param>
        <module name="ConvertToDrilldownSearch">
          <module name="ViewRedirector">
            <param name="viewTarget">flashtimeline</param>
          </module>
        </module>
      </module>
      <module name="ViewRedirectorLink">
        <param name="viewTarget">flashtimeline</param>
      </module>
    </module>
  </module>

[lguinn2]

Here are some ideas:

• Run either one of your searches from the Search window, interactively. When it completes, take a look at the Search Job Inspector. This will give you some information about how long the search runs, which may help you figure out how to do it more efficiently. More info about the Search Job Inspector is here.
• The append command will run by default for only 60 seconds. You might need to increase that by using the maxtime option. Here is the link to the manual entry for append.
• I would consider using post-process in your dashboard. Using this technique, you run one base search that collects all the data and computes all the statistics. The each panel displays the appropriate statistics, computed from the base search. This technique can be used in a variety of situations, but I think it is particularly appropriate here. This is discussed in the documentation under Use one search for a whole view. However, I think you should also download and install the Splunk Dashboard Examples app. You don't need to install it in production, just somewhere where you can review and learn from the examples; it's a very practical and useful resources.

HTH