Does a HiddenPostProcess limit the dynamic dashboard modules?

[inglisn]

Hi

I've been building a dashboard that contains a number of dynamic elements. Two modules are fed by a search that will return anything from a few thousand to a few million events depending on time range. One of these modules is a context sensitive table which triggers some more charts to appear using intentions.

First structure was this:


HiddenSearch
HiddenChartFormatter
FlashChart
HiddenSearch
SimpleResultsTable with drilldown
ConvertToIntention
ConvertToIntention
HiddenChartFormatter
FlashChart


This worked, but because I was running identical base queries at the same time each returning millions of events the performance was poor. It would take over a minute to get anything back and we lost the chart preview building during the wait time.

I swapped this for a single HiddenSearch and two HiddenPostProcess clauses but then the chart triggered by the click-through stopped appearing. I can see in the logs that the intentions are working and the query it generates does what it should do when I run it manually.

I played around with the structure and it seems that either the HiddenChartFormatter or the FlashChart cant live inside a HiddenSearch => HiddenPostProcess structure.

Is there some constraint?

[sideview]

The reason is that the postProcess doesn't get consumed when it's used. So it's still there at the second search for your second chart.

The solution is clunky, but it's to put a second HiddenPostProcess nested in there, with an empty value or space char for the search param. That will clobber the postprocess at that point where you dont want it to apply, and then the results from the second search will render just fine.

It's easy to think that postprocess gets consumed when it's used, like intentions do. However it hangs around like timeranges and search strings do.

[inglisn]

No answers yet, but I did switch to JSChart (no difference) and run splunk in debug mode to have a look at the _internal indexes. There is a repeating pattern around the time that the drilldown is run:

class=js_chart.js, XHR clear for takeoff for module JSChart_1_15_0
class=js_chart.js, XHR in-flight destroyed for module JSChart_1_15_0 for job 1332161165.8 and replaced with new one
class=js_chart.js, Splunk.Module.JSChart .getResults() aborted
class=js_chart.js, Aborting getResults request for Splunk.Module.JSChart

Any ideas why the module would abort the search?

[MHibbin]

inglisn,

A copy of your XML may be of use to SplunkBase, to see if there are any issues in your syntax.

However, have you looked at the following documentation (I came across it this morning, whilst doing my own HiddenPostProcesses)?

http://docs.splunk.com/Documentation/Splunk/latest/Developer/PostProcess

I believe this is what you are looking for. Basically your structure (very basic overview) should be something like this...

HiddenSearch (Module)
Search (Param)
HiddenPostProcess (Module)
Search (Param)
HiddenChartFormatter (Module)
FlashChart (Module)
HiddenPostProcess (Module)
Search (Param)
HiddenChartFormatter (Module)
FlashChart (Module)
AND SO ON...

Hope this helps,

MHibbin

[inglisn]

Here are the relevant parts of the dashboard. Search performance is reasonably and the drilldown triggers searches containing the correct intentions but the chart never appears.

<module name="HiddenSearch">
  <param name="search">index=main sourcetype=rpt-pur-2 filter="foo" | eval dVolGB=(DOWNSTREAM_VOLUME/1000000) | bucket span=1h _time | stats sum(dVolGB) as downGB by _time, date_wday, PackageName, ServiceType | fields + _time date_wday PackageName downGB ServiceType
  </param> 

  <module name="HiddenPostProcess">
    <param name="search">timechart sum(downGB) by ServiceType</param> 

    <module name="SimpleResultsHeader" layoutPanel="panel_row2_col2">
      <param name="entityName">results</param>
      <param name="headerFormat">Selected Packages: Download Activity per day</param>
    </module>

    <module name="HiddenChartFormatter" layoutPanel="panel_row2_col2">
      <param name="chart">column</param>
      <param name="chart.stackMode">stacked</param>
      <param name="primaryAxisTitle.text"></param>
      <param name="secondaryAxisTitle.text">GBytes downloaded</param>
      <param name="legend.placement">none</param>
      <module name="FlashChart">
        <param name="width">100%</param>
        <param name="height">320px</param>
        <param name="enableResize">False</param>
      </module>
    </module> <!-- end chart stuff -->

  </module> <!-- end hidden post process -->    

  <module name="HiddenPostProcess">
    <param name="search">
eval dtg=strftime(_time, "%H")
| stats sum(downGB) as downGBytes by dtg, date_wday, PackageName
| dedup 1 date_wday, PackageName sortby -downGBytes
| chart values(dtg) over PackageName by date_wday 
| fields PackageName sunday monday tuesday wednesday thursday friday saturday
    </param>

    <module name="SimpleResultsHeader" layoutPanel="panel_row2_col1">
      <param name="entityName">results</param>
      <param name="headerFormat">Select Packages: Peak hour per day.</param>
    </module>

    <module name="SimpleResultsTable" layoutPanel="panel_row2_col1">
      <param name="displayRowNumbers">false</param>
      <param name="drilldown">all</param>
      <param name="count">0</param>


      <module name="HiddenSearch">
        <param name="search">index=main sourcetype=rpt-pur-2 NOT "TIME_STAMP" | eval downMBytes=(DOWNSTREAM_VOLUME/1000) | chart sum(downMBytes) by date_hour, ServiceType 
        </param> 

        <module name="ConvertToIntention">
          <param name="intention">
            <param name="name">addterm</param>
            <param name="arg">
              <param name="date_wday">$click.name2$</param>
            </param>
            <param name="flags"><list>indexed</list></param>
          </param>

          <module name="ConvertToIntention">
            <param name="intention">
              <param name="name">addterm</param>
              <param name="arg">
                <param name="PackageName">$click.value$</param>
              </param>
              <param name="flags"><list>indexed</list></param>
            </param>

            <module name="SimpleResultsHeader" layoutPanel="panel_row2_col1">
              <param name="entityName">results</param>
              <param name="headerFormat">$click.value$ : utilisation on $click.name2$</param>
            </module>

            <module name="HiddenChartFormatter" layoutPanel="panel_row2_col1">
              <param name="chart">column</param>
              <param name="chart.stackMode">stacked</param>
              <param name="primaryAxisTitle.text">Hour</param>
              <param name="secondaryAxisTitle.text">MBytes downloaded</param>
              <param name="legend.placement">none</param>
              <module name="FlashChart">
                <param name="width">100%</param>
                <param name="height">320px</param>
                <param name="enableResize">False</param>
              </module>
            </module> <!-- end chart stuff -->

          </module> <!-- end c2i click.value -->
        </module> <!-- end C2I click.value2 -->           
      </module> <!-- end HiddenSearch --> 
    </module> <!-- end simple results drilldown -->
  </module> <!-- end hidden post process-->
</module> <!-- End HiddenSearch (or drilldown table postprocess) -->