Optimize repeat searches in append

[harshal_chakran]

I have a search in below format:

index=xyz sourcetype=abc...|table code...
|join code[search index=def ....]
|where $dd1$="value" AND $dd2$=""|some operations using stats...1
|append [index=xyz sourcetype=abc...|table code...
|join code[search index=def ....]
|where $dd1$="value" AND $dd2$="value" AND $dd3$=""|some operations using stats... 2]
|append [index=xyz sourcetype=abc...|table code...
|join code[search index=def ....]
|where $dd1$="value" AND $dd2$="value" AND $dd3$="value"|some operations using stats...3 ]

I have multiple dropdowns dd1, dd2 and dd3 and based on which search portions 1,2 and 3 will run respectively.
Means if i select dd1 , then first portion will run with stats operation 1, on selecting dd1 and dd2 -stats operation 2 will run ..and so on.

Currently this search is taking long time to proccess as it is running every search portion till where condition. I tried putting where condition after every index/sourcetype name, but still not that helpful.

Is there any way I can shorten/optimize this search.
Any other alternative would be helpful.

[woodcock]

Here is what I would do. Create 2 base searches, one with index=xyz sourcetype=abc ... | addinfo and another with index=def ... | addinfo and at the end of each, have a section like this:

<done>
   <condition>
        <set token="xyz_job_token">$result.info_sid$</set>
    </condition>
</done>

Then you can load the event data in any panel/search with | loadjob $xyz_job_token$ or | loadjob $def_job_token$ as many times and in as many places as you like/need with no additional search penalty (other than RAM to load the events).

[woodcock]

Add a submit button so that it will not search until you click it; in addition to this:

 <fieldset autorun="false" submitButton="true">

Make sure if you have input fields that you have the searchWhenChanged="false" tag set on each of them:

 <input type="dropdown" token="dd1" searchWhenChanged="false">

[rjthibod]

Suggestions from @cusello are correct, but I would suspect there are other things to look at first.

1. Why do the drilldown booleans (e.g., $dd1$="value") not work in the first search term index=def ...? That is going to be one of the best places to fix this if possible
2. Can you use eventstats or appendpipe to get any of the second or third-level stats without searching for events again?
3. Per @sideview, you can often use stats in place of join and append. It will be faster if you can apply that. Check out his .conf talk on this issue.
4. At the very least, you could put the first search index=xyz sourcetype=abc in a standalone search, use features in SimpleXML to get the job SID when that search completes, and then use loadjob to reference those results in the subsequent searches.

It would really help if we had more information about the data. Especially when it comes to the first point, are the fields use for filtering not search-time generated fields, e.g., you have to calculate them in the search.

[jacobpevans]

For (4), you can just declare the standalone search as a base search and refer to it that way. Example:

<form>
  <label>Test Dashboard</label>
  <search id="baseSearch">
    <query>index=xyz sourcetype=abc ...|table code...
</query>
    <earliest>0</earliest>
  </search>
...
   <row>
     <panel>
       <chart>
         <search base="baseSearch">
           <query>$new_search$</query>
         </search>
         <option name="charting.chart">bar</option>
       </chart>
     </panel>
   </row>

That said, I would use the base search, then create full panels for each of the search "options", set basic true/false tokens (instead of changing the search), and then hide or show whichever panel is being requested.

[gcusello]

If I correctly understood: you have three search condition and you would combine them in every possible combination (six).

Maybe you could have best performances creating six searches resulting from the combinations of your searches (probably they use similar conditions) and separately call one of them from your dropdown, in this way you could create optimized search without (or limiting) join and append commands.
Otherwise, if the results of each search are aggregated data (result of stats command, not table), you could use tscollect to accelerate the single searches.
In addition, you don't need to use table command before join (it's another step that slows you search).
Bye.
Giuseppe

[harshal_chakran]

Hi, I am trying to implement @cusello advice :

  <fieldset autoRun="true" submitButton="true">
    <input type="dropdown" searchWhenChanged="false" token="token_header">
      <label>Header</label>
      <choice value="*">All</choice>
      <search>
        <query>search to load dropdown|table header</query>
      </search>
      <fieldForLabel>header</fieldForLabel>
      <fieldForValue>header</fieldForValue>
      <default>*</default>
      <change>
        <condition label="All">
          <set token="new_search">index=xyz sourcetype=abc...|table code...|join code[search index=def ....]|where $dd1$="value" AND $dd2$=""|some operations using stats...1</set>     
        </condition>
        <condition match="$value$!=&quot;*&quot;">
          <set token="new_search">index=xyz sourcetype=abc...|table code...|join code[search index=def ....]|where $dd1$="value" AND $dd2$="value" AND $dd3$=""|some operations using stats... 2</unset>
        </condition>
      </change>
    </input>
  </fieldset>
  <row>
    <panel>
      <chart>
        <search>
          <query>$new_search$</query>
          <earliest>0</earliest>
        </search>
        <option name="charting.chart">bar</option>
      </chart>
    </panel>
  </row>

Above scenario is for first dropdown only, where if I select "All" - search with stats 1 will set in token "new_search" and if any other value is selected -search with stats 2 will set , and build the graph.

What happens here is chart starts loading after dropdown selection and not after Submit button. Even though I have kept searchWhenChanged="false".
Is there any way I can load the panel after submit button click?

[gcusello]

Don't set a default in your dropdown it's the easy way, dashboard has to wait for your choice.
Bye.
Giuseppe