Why does my dashboard search never complete and dies after one hour? Is this a TTL issue?

[gabriel_vasseur]

I am really confused by TTLs. My understanding from what I've read is that it's 10 minutes by default, and 7 days if you save the job or send it in the background.

But what happens with dashboards? I haven't seen anywhere I can change the TTL, either as a user looking at the dashboard or as a developer in Simple XML or elsewhere.

I have a dashboard that runs on a lot of data and understandably takes a long time to run. Eventually, I want to use an accelerated data model to make it quicker, but for now it's not an option. I want to schedule a PDF, so I don't care if the search takes hours. However, looking at the "activity/jobs" the job disappears after one hour. Not 10 minutes, not 7 days, one hour. Confused!

It's frustrating too because it looks like the search would need just over an hour to complete...

[jkat54]

Using limits.conf (which must be on the indexers in this case):

[search]
ttl=<integer>

http://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf

This is how long the search is kept on disk AFTER it completes. So it's not the issue.

Could you be hitting a web time out? as in persistence through load balancer time outs, auth token timeouts, etc? The default auth token timeout is 1 hour.

Also in limits.conf but should be applied to search heads instead:

[authtokens]
expiration_time = <integer>
* Expiration time of auth tokens in seconds.
* Defaults to 3600

[jkat54]

Just to note. The auth token has a limited life that starts when you log in and doesn't renew itself as you click around in the app. If you set it to default of 8hrs then you'll get kicked out of the app exactly 8hrs after first login. Closing this question now.

[gabriel_vasseur]

Authentication I believe was not a problem, because I was still logged in doing other things.
Following your suggestion, I did some grepping and I'm thinking maybe the problem could be fixed by tweaking the "dispatch.ttl" value. It's set as "2p" by default, which is only going to mean something for scheduled searches, which my dashboard wasn't. Anyway, now I'm moving to a summary search system anyway, so I won't be trying.

[jkat54]

Should we close this question then?

[somesoni2]

You could explore option of summary indexing to improve your search performance OR (since it's PDF delivery), split the search and saved the results in lookup table and update your dashboard to use the result from lookup table. If you can share your search, we can see if there is any scope of improving the performance.

[gabriel_vasseur]

Thanks somesoni2. I eventually did the temporary lookup table trick, which allowed me to produce the report. I am now exploring the summary index solution so it can be automated properly (see another question of mine!). I believe the search itself is fine, it's just that it has to run over many millions of events. I just wondered why it would die after one hour. I guess Splunk moves in mysterious ways!