Realtime Cached Charts In Web Framework

[aelliott]

I have caching turned on for my searches within a dashboard using Django and Web Framework and when I switched my time picker to realtime within an amount less than the cache time, it appears to continuously run the queries even though I have completely exited the dashboard. These queries will run forever until manually stopped since they will never elapse their cache time. I do believe this is a bug.

Here is a small sample. in the timerange dropdown, select a minute to see issue.

  {% timerange id="activitysummary-timerange" preset="Last 30 days"  %}

                        {% chart
                    id="sizesegment-piechart" managerid="sizesegment-piechart-search"
            drilldown="all"
            drilldownRedirect=False
            resizable=True
                    type="pie" %}
{% searchmanager
    autotstart=True
    id="sizesegment-piechart-search"
    search="index=myIndex CallingApplication=DP  | top limit=10 SizeSegmentID"|token_safe
    preview=True cache=120 %}

Then in javascript. 
 activitysummarytimerange.on("change", function () {
        sizesegmentpiechartsearch.search.set(activitysummarytimerange.val());
    });

So cache is set to expire in 120 seconds.. but since the search includes the realtime of 60 seconds.. the search refreshes itself and will expire in 120 seconds.. then 60 seconds later refreshes itself.. and will be set to expire in 120 seconds.. Even after leaving the dashboard this will continue.
The expiry date gets reset to 120 seconds later every 60 seconds.

Alternatively you could set the realtime to above 120 seconds...
So cache is set to expire in 120 seconds.. since the search includes the realtime of 140 seconds.. It will expire after 120 seconds and refresh in 140 seconds which will then refresh the cache.. as long as you are on the dashboard.

So one expires.. and the other does not.
If i go into activity -> jobs.. here is what i see after several minutes after leaving the dashboard.. cache=120 and time dropdown was set to realtime 30 seconds..
Dispatched at: 1/21/14 3:06:23 PM
Owner: myself
Size: .35MB
Events: 0
Run time: 00:17:31
Expires: Jan 21, 2014 3:33:54 PM
Status: Running(100%)

if i refresh the screen.. i now see the following:
Dispatched at: 1/21/14 3:06:23 PM
Owner: myself
Size: .35MB
Events: 0
Run time: 00:18:21
Expires: Jan 21, 2014 3:34:45 PM
Status: Running(100%)
The Expiry date gets later and later and the run time gets longer and longer. for something that should have expired after 120 seconds.

Now I've gone back into the dashboard and set the realtime amount to 60 seconds(cache is still 120), left the dashboard and waited a few minutes.. now every search is showing up twice in there.. to run forever.

Have gone back in the next morning and the one with cache=120 and 30 seconds realtime is up to 17 hours.

[ineeman · Accepted solution]

@aelliott: I finally had a chance to play around with this, but I cannot seem to reproduce a bug. That said, let me describe what the expected behavior is, and you can tell me if this is what you expect and also if I'm describing something relevant to your issue.

When you set cache=X (where X is a non-negative integer or false) on a SearchManager, it has the following effects:

1. When a search job is going to be created (e.g. when the query change), then we will first look to see if there is a previous search job that matches all the properties. If there is one, then we will look to see if it is "fresh" enough, where "fresh" is defined as the job being created less than X seconds ago (if it is a non-negative integer) or any job (if X===true). If X===false, we will always create a new job. For SavedSearchManagers we also have the cache="cached" option, which just says to look for previously scheduled instances of this saved search.

2. If cache is "turned on", then we will never automatically cancel the search job. Usually, we will cancel the search job in two cases: (a) if a new search job needs to be created (e.g. when the query changes), or (b) if you navigate away from the page (if the search is complete). When caching is turned on, we do not do either of these. The reason is relatively simple: imagine you have a dashboard with a cached search, and you load the dashboard in two tabs (or two different users load it). Both of them will load the same search job, and if we cause a cancellation of the search, we've suddenly created a jarring experience for one of the tabs/users.

Given that, it makes sense that when a real time search is set with cache=True we will not cancel it on page navigation (or even when the search changes). In reality, for real time searches, we do not cancel them altogether, as they are never 'done'. Note that even the main search page (in the search app) does not cancel real time searches (it pauses them).

In your above sample code, if you change the the cache=120 to cache=False, you'll notice that when you change it to a real time search and then back to a non-realtime search, the real time search is cancelled. However, on page navigation, the search will continue running (per the notes above).

Finally, I've included below the full test page I used. It's very possible I misunderstood your issue, and if so, just let me know and we can re-work through it. Let me know if you have any other questions, and thank you for the feedback!

---

{% extends "splunkdj:base_with_app_bar.html" %}

{% load splunkmvc %}

{% block title %}Itay's Test Page{% endblock title %}

{% block css %}
    <link rel="stylesheet" type="text/css" href="{{STATIC_URL}}splunkjs/css/dashboard.css" />
{% endblock css %}

{% block content %}

<div class="dashboard-body container-fluid main-section-body">
    <div class="row">
        <div class="span12 dashboard-header clearfix">
            <h2>Test Page</h2>
        </div>
    </div>
    <div class="dashboard-row">
        <div class="dashboard-cell" style="width: 100%;">
            <div class="dashboard-panel">
                <div class="dashboard-element">
                    <div class="panel-head">

                    </div>
                    <div class="panel-body">
                        {% timerange id="activitysummary-timerange" preset="Last 30 days"  %}

                        {% chart
                            id="sizesegment-piechart" managerid="sizesegment-piechart-search"
                            drilldown="all"
                            drilldownRedirect=False
                            resizable=True
                            type="pie" 
                        %}

                    </div>
                </div>
            </div>
        </div>
    </div>
</div>

{% endblock content %}

{% block managers %}
    {% searchmanager
        autotstart=True
        id="sizesegment-piechart-search"
        search="index=_internal | head 1000 | top limit=10 sourcetype"|token_safe
        preview=True cache=120 %}
{% endblock managers %}

{% block js %}
<script>
require(["splunkjs/ready!"], function(mvc) {
    var timerange = mvc.Components.get("activitysummary-timerange");
    var manager = mvc.Components.get("sizesegment-piechart-search");

    timerange.on("change", function () {
        manager.search.set(timerange.val());
    });
})
</script>
{% endblock js %}

[ineeman]

The cache property is really only used for fetching the job - it has no impact on search creation. A real time search has no end, and thus will never finish on its own (something needs to cancel it). We can do a couple of things:
1. Pause a real time search on page navigation (this is what the search page does)
2. Set the auto_cancel property on the real time search to the same as the cache value.
Happy to hear more suggestions.

[aelliott]

Here would be my expected behavior:
The realtime search runs while the dashboard is open.
The cache is kept for it's duration.
What happens when you leave?
The realtime search only runs until the cache runs out of time.
I think of selecting a realtime search in a dashboard.. being connected to that dashboard, same with the cache.. so if none of that dashboard is open, then the realtime search should not be running, and the cache should not be updating.

[aelliott]

The auto_cancel works great! I have 7 different tokens that can be set by the user and setting it to realtime and setting those tokens at once could make splunk fill up with realtime searches that would run forever really fast. Thanks for your assistance.

[ineeman]

If you want the RT search to cancel after some period of inactivity, just set the auto_cancel parameter on the manager. You can see it in action here: http://docs.splunk.com/Documentation/Splunk/latest/RESTAPI/RESTsearch#POST_search.2Fjobs (note that while on the page your search won't cancel as we ping it).

(in two comments due to character limits)

[ineeman]

When do you expect it to cancel? A real time search will continue running even when you navigate away from the page, due to the fact that it is never done (as I noted above). Setting cache=False will just mean that if you change a setting on the manager (e.g. change the query) we will cancel the previous search.

[aelliott]

Ok so i tested what you said above. set the cache=120, set realtime to 30 seconds.. they ran all night.. So i changed the cache=False.. and set the realtime to 30 seconds.. same exact parameters.. and they are still running...

[ineeman]

Not yet - things went a bit haywire due to the holidays, will try and get to it in the next couple of days!

[aelliott]

Was this determined a bug? Were you able to replicate?

[aelliott]

added to description, a realtime search must be selected in the timerange dropdown less than the value of the cache timeout to see this issue.

[ineeman]

Can you post a sample search and chart definition? I want to make sure we can replicate your specific issue.