All Apps and Add-ons

Sideviews - pass several searches output downstream

splunk_zen
Builder

Nick,
anyone else,

how can I use the ResultsValueSetter (or other similar module) to pass the various outputs of several different parallel Search modules so I can pipe them into a final Search module ?

I have some considerably large searches which I need to concatenate into a final expression (each output has its own weight in it)

I currently have a search output which is being fed to other downstream searches,

  <module name="Search" layoutPanel="panel_row1_col1" autoRun="True">
    <param name="search">
    ....
    | fields date_month, date_mday, date_hour as BusyHours
    </param>

      <module name="ResultsValueSetter">
        <param name="fields">BusyHours</param>

    <module name="Search" layoutPanel="panel_row2_col1">
        <param name="search">
        source=... $BusyHours$
        | a_lot_of_conditions_and_processing
        | stats sum(score1) as score1
        </param>

        <module name="HiddenChartFormatter">
            <param name="charting.chart">radialGauge</param>
            <param name="charting.chart.rangeValues">[0,50,70,85,94,100]</param>
            <module name="FlashChart" />
        </module>

    </module>

    ( rinse and repeat....)

    <module name="Search" layoutPanel="panel_row6_col1">
        <param name="search">
        source=... $BusyHours$
        | more_conditions_and_processing
        | stats sum(score1) as score5
        </param>

        <module name="HiddenChartFormatter">
            <param name="charting.chart">radialGauge</param>
            <param name="charting.chart.rangeValues">[0,50,70,85,94,100]</param>
            <module name="FlashChart" />
        </module>

        </module>        
    </module>

However, I need to get a weighted composed indicator that represents the sum of those parts,

| eval ComposedScore=100*(0.12*$score1$ + ... + 0.17*$score5$)

How can I save those intermediate results as keys to feed them to that final search ?
EDIT
Let's say my main search returns the BusyHours,
my 2nd level searches compute some % Efficiency stats (over different sources),
my 3rd level search should return a weighted score (based on contributions from the 2nd level ones)
I've some gauge and line charts over the 2nd level searches
but I'm failing to access the results from the 2nd tier searches to generate the final composed charts.
I'm having trouble with the XML hierarchy data flow,
this is my current structure,

<module name="Search">
<param name="search">
# main search
</param>

  <module name="ResultsValueSetter">
    <param name="fields">BusyHours</param>

    <module name="Search" layoutPanel="panel_row1_col1">
      <param name="search">
      # first 2nd level search
      </param>

 <module name="PostProcess">
    <param name="search">
| stats sum(MemScore) as MemApdexScore
</param>

    <module name="HiddenChartFormatter">
      <param name="charting.chart">radialGauge</param>
      <module name="FlashChart" />
    </module>
</module>

    <module name="Search" layoutPanel="panel_row2_col1">
      <param name="search">
      # second 2nd level search
      </param>

 <module name="PostProcess">
    <param name="search">
| stats sum(CpuScore) as CpuApdexScore
</param>

    <module name="HiddenChartFormatter">
      <param name="charting.chart">radialGauge</param>
      <module name="FlashChart" />
    </module>
</module>

    <module name="Search" layoutPanel="panel_row2_col1">
      <param name="search">
      # third 2nd level search
      </param>

 <module name="PostProcess">
    <param name="search">
| stats sum(Service1Score) as Service1ApdexScore
</param>

    <module name="HiddenChartFormatter">
      <param name="charting.chart">radialGauge</param>
      <module name="FlashChart" />
    </module>
</module>

How can I get these objects data outside of their respective ?

| eval TotalScore = $MemApdexScore$ + $CpuApdexScore$ + $Service1ApdexScore$

sideview
SplunkTrust
SplunkTrust

Can you add more details? Have you already looked at the examples and documentation in "Module Documentation > Advanced Modules > The ResultsValueSetter Module"?

Admittedly that docs page has a bug where the page has no scrollbar (sorry - it'll be fixed in the next release), but it gives a good intro to ResultsValueSetter nonetheless.

If you need to format the various rows of key-value pairs into a boolean search expression, the best way is to use Splunk's format command to do that work before using ResultsValueSetter to pull down the formatted string.

UPDATE: I don't think ResultsValueSetter is what you're looking for at all. Actually I think you're just looking for search-language help. Possibly you need a subsearch but it's far from clear to me that you do.

Looking at your search, your | fields date_month, date_mday, date_hour as BusyHours looks like you're getting the "busy hour" for each day, perhaps for a phone system or the like.

So if coming out of that search, each row represents a particular day, and each row has a field called BusyHours that represents the busy hour for that day, then you should be simply piping those rows into your | a_lot_of_conditions_and_processing

From there, to render N different radial gauge charts, you can use PostProcess, so that you have several blocks that look like this:

<module name="PostProcess">
  <param name="search">stats sum(score1) as score1</param>
  <module name="HiddenChartFormatter">
    <param name="charting.chart">radialGauge</param>
    <param name="charting.chart.rangeValues">[0,50,70,85,94,100]</param>
    <module name="FlashChart" />
  </module>
</module>

All of those postProcess searches can hopefully run against the same base search-results set, and this can be a very good thing in terms of efficiency. However postProcess is a pretty complex topic and it can be quite confusing. I would make sure to start by reading the "postprocess intro" in Sideview Utils as this will help you avoid the common pitfalls, and remember that search commands just operate on rows of key value pairs, and that the search commands other than that are dumber than you might think -- you can usually debug a complex search problem just by looking at what fields and what rows are actually coming out at each successive pipe and visualizing what the next search command would do with those rows.

UPDATE 2: Ah! I think I see your question finally. For score 1 you have to wash the rows through some search language and then into a simple stats clause generating score 1. For score 2 you have to wash the rows through some very different search language and then generate score 2. Etc... It sounds like you cannot (or think you cannot) run all the processing on a single set without creating a horrendous trainwreck - ie in the process of calculating score1 you have to damage something that you need to generate score 2.

If that's the case, then I still recommend postProcess, but you have to kind of nest things inside eachother and use several different ResultsValueSetters to pull each layer of single values down. Remember always that ResultsValueSetter will only ever get field values from the first row.

So here's the basic idea.

Search 
  (your base search here)
  PostProcess
    (pipe the rows through the processing for `score1` and end up with a single row, and a `score1` field. 
    ResultsValueSetter
      `<param name="fields">score1</param>`
      PostProcess
        (pipe the rows through the processing for `score2`, end up with a single row, single `score2` field
        ResultsValueSetter
          `<param name="fields">score2</param>`
          ...   so on and so forth, you keep nesting. 


                    Search
                      aggregateScore = $score1$ * $score2& + 10*($score3) 

etc... If you don't explicitly reference $postProcess$ in a postprocess search, then each PostProcess module will just clobber the postProcess search of the ones upstream. So each postprocess here is running directly against the base search results

If you post the full details of all the processing in the search language (probably best as a different question as this is getting super long), then I still think you might be able to finagle all the processing into a single pipeline. Or we can steer you in the direction of tricks that could allow you to do at least fewer layers of PostProcess+ResultsValueSetter.

0 Karma

splunk_zen
Builder

"All of those postProcess searches can hopefully run against the same base search-results set"
That is the problem, only 2 of the 8 searches are over the same sources.
It seems I'm still missing something, I get the PostProcess flow pipe analogy but can't I really access simultaneously different search-results sets, that is, the data at different pipe endings ?
I've tried converting the XML to use PostProcess but I'm still running into the downstream nature of the searches.
I've edited the question to illustrate how I need the data.

0 Karma

splunk_zen
Builder

I am sorry if I was not clear enough.

I am piping the BusyHours key into each of the downstream searches:

index=resources_reporting source="*/output/ora_queries/client04/OS_metrics.csv" $BusyHours$

| a_lot_of_conditions_and_processing
| stats sum(score1) as score1

it is working correctly with the ResultsValueSetter as the first search output is only one line (a series of OR dates).

0 Karma

splunk_zen
Builder

I had read the complete Sideview tutorial, but it seems I'm missing something on its mechanics.
I hope my question edit brought some light to what I'm expected to achieve.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...