Is it efficeint to use lookups to save historic data?

[glenngermiathen]

I have several searches I use to trend historic data, however they take a long time to complete. The data is historic so it doesn't change, but will have the current data added to it. I wanted to see if anyone had tried or had thoughts about saving the data to a lookup with outputlookup, and then using that for future searches?

[micahkemp]

My overly simplified summary:

Lookups are for state and/or event enrichment

Summary indexing is for making your data more quickly searched later, when you pretty much know which pieces of the events you need to search/report on at that time. You can think of them as pre-computed segments of a search you want to run later.

[elliotproebstel]

As suggested by @gpradeepkumarreddy above and by your post, the two most likely competing options for making these historical searches run more efficiently would be to either create a lookup that is periodically updated or to use scheduled searches to generate summary indexing. Figuring out which option is best for you will likely require some testing. I've used both for varying use cases, and here are some lessons I've learned along the way:

Lookup files at a small scale can be very efficient for accessing statistical trends, but they live on the search head (or search head cluster) and can be wiped out with a single errant | outputlookup command. Unless you have an external backup mechanism in place, you can lose all that pre-calculated data from a poorly written search command on Monday morning before you drink your coffee. Additionally, as the lookup files grow, they make your search bundles grow. At a few thousand lines, this probably won't be a big deal; at a few million, they can impact performance across the board.

Summary indexes will, of course, live on your indexers. You can't use them in a lookup, but they are protected from accidental erasure. They will consume disk space on your indexers, but they won't consume license if configured properly in alignment with documentation:
http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Knowledge/Usesummaryindexing

Ultimately, your best bet is to try both on a small scale and see which best meets your needs and protects you from yourself.

[glenngermiathen]

Thanks for the lessons learned. I am following your advice and doing some small scale testing to see which works best, but I think it will be a summary index.

[elliotproebstel]

Great! One other word of warning about using a lookup table (even though you're leaning away from it) - because a lookup table lives on your search head, it's not shared with other search heads unless they are clustered. So you can't build it on one standalone search head and use it on another. Likewise, if you need to replace that search head, you'll have to manually move knowledge objects, including that lookup table, to your new search head. Just documenting those here for others who come along and are weighing their options.

[pradeepkumarg]

Did you try summary indexing ?
http://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Knowledge/Usesummaryindexing

It's similar but you write to an index instead of a lookup and use summary index for your reporting searches.

[glenngermiathen]

I am reviewing this approach now. Thanks for the suggestion!.