Successor Search Using Previous Saved Search Result

[jackyc]

Hi there,

I am constructing a series of searches for a dashboard for annual audit. Because it is necessary to parse out the un-used portion of the maillog, is it possible to use a search first does this and saved the result, then designed other searches based on this result? In this case, maybe call it base search. If the base search is scheduled, it can save the resources which the successor search need to run for the base search. Is that kind of search templates? Any idea for discussion.

Many thanks.

Br, Jacky.

[Ayn · Accepted solution]

Not entirely sure what you're wanting to achieve, but if I understand you correctly you want to run a search first of all for gathering base information that you can use in other searches, and thereby making it much faster to run those searches than if they were run against the full "raw" data? If that's what you want, this is achieved through "summary indexing" in Splunk and is documented here: http://www.splunk.com/base/Documentation/4.1.6/Knowledge/Usesummaryindexing

What you'll want to do is schedule a search to run with some interval and have it save its results to the summary index, like for instance a scheduled search with the name "Summary - top mail senders":

sourcetype="maillog" | sitop sender

This will make Splunk save the results from this query to the summary index. To grab stats from the summary index you can then do

index=summary search_name="Summary - top mail senders" | top sender

which will speed up things a whole lot compared to running the same search against the default index as it will only need to work on the aggregated data in the summary index.

[gkanapathy · Accepted solution]

On a dashboard, you can use the searchPostProcess in Simple XML http://www.splunk.com/base/Documentation/latest/Developer/FormSearchPostProcess, or HiddenPostProcess in Advanced XML. If you're looking at just running searches, you can run a search, then use the | loadjob command to retrieve the results of another previously-run search and pipe those to another command.

Do be aware that in both cases, you must make sure that the base search results do not optimize away fields you'll want to postprocess later. You can use the fields, stats or table commands to make sure the needed fields are retained in the base search results.

[jackyc]

Thx, gkanapthy. In compare of the Summary Indexing, this approach seems much light weight. What it does is simply pipe the previously-run search result. If the search is not a saved search, I think the loadjob command would fire previous search on the fly, right?

[Ayn · Accepted solution]

Not entirely sure what you're wanting to achieve, but if I understand you correctly you want to run a search first of all for gathering base information that you can use in other searches, and thereby making it much faster to run those searches than if they were run against the full "raw" data? If that's what you want, this is achieved through "summary indexing" in Splunk and is documented here: http://www.splunk.com/base/Documentation/4.1.6/Knowledge/Usesummaryindexing

What you'll want to do is schedule a search to run with some interval and have it save its results to the summary index, like for instance a scheduled search with the name "Summary - top mail senders":

sourcetype="maillog" | sitop sender

This will make Splunk save the results from this query to the summary index. To grab stats from the summary index you can then do

index=summary search_name="Summary - top mail senders" | top sender

which will speed up things a whole lot compared to running the same search against the default index as it will only need to work on the aggregated data in the summary index.

[jackyc]

Thx, Ayn. You are exactly talking what I am going to do. The summary index is very interesting, I think I will look deep to it and see what else it can do. But I think if the splunk will create a new index and this approach would fit for more complicated series of searches. By the way, will the summary index consumes extra space and license volume? In compare of loadjob command, I think summary index would be some how more complicated.