Splunk Search

Does a sorted lookup table result in faster lookups?

hulahoop
Splunk Employee
Splunk Employee

Am curious what the performance difference is between sorted and unsorted lookups (sorting by the primary search key of course), or if there is any.

Tags (3)

supersleepwalke
Communicator

Lookups are only indexed over a certain size:

http://splunk-base.splunk.com/answers/8326/are-lookup-tables-indexed

http://splunk-base.splunk.com/answers/10160/at-what-point-do-very-large-lookup-files-csv-get-indexed

I tested using a sorted and then randomized lookup table. The results are below.

Sorted lookup table:

Duration (seconds)      Component       Invocations     Input count     Output count
147.425                     command.lookup  302             104             104

Randomized lookup table:

Duration (seconds)      Component       Invocations     Input count     Output count
199.059                     command.lookup  301             104             104

Caveats:

  • The original lookup table was created using outputlookup
  • The random version was created using sort -R
  • For some reason, my Splunk instance has NOT indexed these lookup tables, even though they are 15 MB (over the limit)
  • Both files are gzipped
  • I ran the test several times, results were always close this output

araitz
Splunk Employee
Splunk Employee

Lookups are indexed (either in memory or on disk), so I doubt that there is any advantage to presorting the CSV file.

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...