ResultsValueSetter values being lost

[phoenixdigital]

I was hoping someone might be able to shed some light on an issue I have encountered recently which has been a bit frustrating to resolve.

I have an advanced XML dashboard which performs a range of nested searches (about 8 searches) which each populate some variables with ResultsValueSetter.

The final set of results then populates a HTML module to display all the results set by ResultsValueSetter to the user.

The issue I have come across is that after a particular ResultsValueSetter module all the previously set variables then disappear.

The other weirdness is that the dashboard is sometimes inconsistent between a windows and Unix version of Splunk.

So below is a snippet of the dashboard XML where it occurs. During the debugging process I put in two HTML modules to track the variables set by ResultsValueSetter.

The first HTML Module (for debug) shows the $cud_company$ which was set correctly at an earlier ResultsValueSetter (from a previous search). However after the subsequent ResultsValueSetter the value of $cud_company$ (and about 20 other variables set) dissapears.

<!-- Search 5 to determine the Invoice Details and also pull in any comments if they exist -->
    <module name="PostProcess" group="Invoice Details">
        <param name="search"><![CDATA[

            `report_invoice_details($Mirn$,$Year$,$Month$,$reportIn_Total$)`

        ]]></param>

        <!-- TESTING MODULE FOR TRACKING ISSUE OF LAST ResultsValueSetter losing all data -->
        <module name="HTML" layoutPanel="panel_row1_col1">
            <param name="html"><![CDATA[

                <h1>cud_company - $cud_company$ and FyTotal = $FyTotal$</h1>

            ]]></param>
        </module>

        <module name="ResultsValueSetter">
            <param name="fields">FyTotal</param>

            <!-- TESTING MODULE FOR TRACKING ISSUE OF LAST ResultsValueSetter losing all data -->
            <module name="HTML" layoutPanel="panel_row1_col1">
                <param name="html"><![CDATA[

                    <h1>cud_company - $cud_company$ and FyTotal = $FyTotal$</h1>

                ]]></param>
            </module>

I will be rebuilding the entire dashboard in stages from scratch putting in each search until I see it break in the next few days but thought I would post this up to see if anyone else can see any glaring error here.

Is there a limit in the number of nested elements with advanced XML?

Is there a limit to the number of subsearches you can perform in Splunk?

[phoenixdigital · Accepted solution]

OK I tracked it down and it was not ResultsValueSetter. It was the search.

<module name="Search">
    <param name="search"><![CDATA[

        sourcetype=somedata fred=$this$ earliest=`date_find_start_of_financial_year("$Year$", "$Month$")` latest=`from_month("$Year$", "$Month$", "")` Energy=* 
        | dedup _time | eval _time = _time-(3600*6) | eval date_hour = strftime(_time, "%H") | eval date_monthNum = strftime(_time, "%m") | eval date_month = strftime(_time, "%b") | eval date_year = strftime(_time, "%Y") | eval date_day = strftime(_time, "%d") 

        | eval stripTimeDayOnly = strftime(_time,"%d %B %Y") 

        | eventstats sum(Energy) as Consumption, avg(co2) as Scope1 by stripTimeDayOnly 
        | eval dayValue = Consumption  * Scope1 / 24 
        | stats sum(dayValue) as monthlyTotal by date_year, date_monthNum 
        | stats sum(monthlyTotal) as FyTotal 
        | fields FyTotal

    ]]></param>

Sure there is some double handling of stats there but the search works when pasted directly into a Splunk search.

However it wont work in the Search Module... unless

you change

    | stats sum(monthlyTotal) as FyTotal 
    | fields FyTotal

to

    | stats sum(monthlyTotal) as FyTotal | fields FyTotal

Then its fixed!!

Go figure I have no idea why it breaks only at that point. But newlines are the culprit, well at least some newlines.

When I get the time I will try to make a reproducable bug and submit it to Splunk. So the spotlight is off SideViewUtils, it was innocent of this crime. The reason this last ResultsValueSetter killed all previous ones was because the search calling it had failed. Failed silently mind you. Nothing obvious in splunkd.log that I could find.

Will post again if I can reproduce in a simpler dashboard.

[phoenixdigital]

As Nick pointed out to me the search in my original post was not actually the one dying it was an inline one further down that was causing the issue.

[phoenixdigital]

I did initially think it might have been the *s as well. But these did not seem to affect it. I tested this previously and it made no difference to the issue.

I will try moving this to a publicly visible Splunk instance so you can see it in action Nick. Maybe later today or early next week.

So the first HTML module in the example above correctly displays $cud_company$ then the ResultsValueSetter (without a *) is performed which should set $FyTotal$. The very next HTML module now has no value for $cud_company$. Also all previously set ResultsValueSetter variable are "cleared" as well.

[sideview]

I created a testcase with lots of heavily nested Search and ResultsValueSetter modules and I can't reproduce any problem. I did have one idea concerning the <param name="fields">*</param> in a couple of these more nested ResultsValueSetter modules. * works fine there, and ResultsValueSetter will obediently go and get "all of the fields" from the results. It's possible that there are fields by these names, maybe even weird autogenerated emptystring-valued ones that you dont know are even there.

Can you replace those *'s with fixed field lists to troubleshoot?

[sideview]

Can you describe any details about when and where the keys are present and when they disappear? I can infer quite a lot from the comments and debugging HTML modules in the XML, but more detail might help. Which HTML fails to display the upper tokens, do all of the upstream RVS tokens disappear or just some, and does it work initially and then start failing later?

I will start going the other way and constructing a test case nesting Search+RVS modules inside eachother.

I don't know of any problems/limitations here fwiw. I'll update with an answer or comment when I can reproduce.

[phoenixdigital]

Thanks for the suggestion Nick I did have multiple offending autoRun="True"

Removing them sadly did not resolve the issue. I have emailed you at the support address as I would prefer not to post up the entire dashboard.

Thanks

[sideview]

My first thought is that you have an autoRun="True" attribute on a module nested inside another module with an autoRun="True". This is just the kind of bizarre behavior that that core Splunk bug can trigger. Can you pastebin the entire XML file or email it to me at support [at] sideviewapps.com ?