Splunk Search

How can I cache search results to avoid data recalculation?

RiccardoV
Communicator

Hi,
I have a dashboard with parameterized search (it takes three arguments from timepicker and dropdowns) that takes long time if I have a big amount of data because of two external lookups I use.

1) Is there a way to cache / save / speed-up my search? Please note that I need realtime results too, because the user can choose real time window too, I cannot use summary index because I don't need just statistical data, but fields like

name category country timestamp

2) Is there a way to avoid the recalculation of the results when I navigate away and back to the dashboard?

thanks!

0 Karma

cpt12tech
Contributor

Export to a lookup table

Then run your dashboard searches from the lookup table.
http://blogs.splunk.com/2011/01/11/maintaining-state-of-the-union/

0 Karma

musskopf
Builder

You could use summary indexes, just use "stats" command at the end add lots of "by clause", like:

search bla | stats count by name, category, country, timestamp

But remember that at summary index will only improve the performance if the data set is smaller.

You might get better performance creating different summary indexes, one for each report or at least grouping things together where it makes sense... also, if you'll be using "by _time", consider using a | bucket span=1h _time (or any other time span) before your stats command to group things a bit.

RiccardoV
Communicator

Hi musskopf and thanks for your reply.
I think that I can't use summary index here because my search is parameterized with two user inputs (sourcetype and field to lookup). So I cannot foresee which inputs the user will use! How can I set up a summary index if I don't know the actual query in advance?

0 Karma

musskopf
Builder

Sure, there are some situations where you need to know better what the report is supposed to return. In my case I do have input fields, for example, country, sex and age.

If you let the user chose any of the fields it won't help. Might be better understand what are the more common combinations to produce one fast report and another very flexible but slow. Btw, are you searching over all indexes as well? limiting the search to a specific index always help.

0 Karma

RiccardoV
Communicator

I'm searching on a single index and on the sourcetype that user will set on rutime

0 Karma

jrodman
Splunk Employee
Splunk Employee

"Report Acceleration" is precisely storing the search-peer answers to a particular class of search, but it's quite tricky to apply this to a dashboard where the search gets parameterized. We would need to have exactly equivalent fully qualified searches to the paramaterized version of the searches in order to know that we can correctly produce the answers ahead of time and safely use them.

Another approach is to simply schedule the searches ahead of time for your dashboard, but again this involves computing the parameter-space that will be used ahead of time.

We don't have a lazy-reuse option. Typically that's not desirable because the answers for searches may change over time, so simply doing something like lazily cacheing answers to ad-hoc searches would be unhelpful.

How to most efficiently get your results is kind of particular to the goals of your searches. For example if you just want to compute reports on a relatively small set of field values , data model acceleration may be the ideal path.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...