Why do saved search jobs disappear?

[terryjohn]

I have several saved searches and they give good results. The problem I have is that they disappear before I expect them to. In Settings -> Searches, reports, and alerts I have set the expiration to 7 days but the jobs typically last a few hours or sometimes less.

I am running Splunk 6.2.3

Can anyone explain what is happening and what I need to do to fix it?

[jtrucks · Accepted solution]

A scheduled search is only stored, by default, in dispatch for twice the length of time between scheduled runs.

Therefore, if you schedule the search every hour, by default, it will retain results for two hours only.

For details, see the following from http://docs.splunk.com/Documentation/Splunk/latest/Admin/Savedsearchesconf:

dispatch.ttl = <integer>[p]

• Indicates the time to live (in seconds) for the artifacts of the scheduled search, if no actions are triggered.
• If the integer is followed by the letter 'p' Splunk interprets the ttl as a multiple of the scheduled search's execution period (e.g. if the search is scheduled to run hourly and ttl is set to 2p the ttl of the artifacts will be set to 2 hours).
• If an action is triggered Splunk changes the ttl to that action's ttl. If multiple actions are triggered, Splunk applies the largest action ttl to the artifacts. To set the action's ttl, refer to alert_actions.conf.spec.
• For more info on search's ttl please see limits.conf.spec [search] ttl
• Defaults to 2p (that is, 2 x the period of the scheduled search).

[terryjohn]

Thanks. By the nature of the problem it's taken me some time to analyse. I agree that should be the answer. Setting the dispatch.ttl = 432000 to all the affected searches worked for some of them but not for others.

I'll have to try to work out why that is.

[terryjohn]

Nothing conclusive but all the search attributes are the same and all searches work fine now. - Thanks for your help