Splunk Search

Does a sorted lookup table result in faster lookups?

hulahoop
Splunk Employee
Splunk Employee

Am curious what the performance difference is between sorted and unsorted lookups (sorting by the primary search key of course), or if there is any.

Tags (3)

supersleepwalke
Communicator

Lookups are only indexed over a certain size:

http://splunk-base.splunk.com/answers/8326/are-lookup-tables-indexed

http://splunk-base.splunk.com/answers/10160/at-what-point-do-very-large-lookup-files-csv-get-indexed

I tested using a sorted and then randomized lookup table. The results are below.

Sorted lookup table:

Duration (seconds)      Component       Invocations     Input count     Output count
147.425                     command.lookup  302             104             104

Randomized lookup table:

Duration (seconds)      Component       Invocations     Input count     Output count
199.059                     command.lookup  301             104             104

Caveats:

  • The original lookup table was created using outputlookup
  • The random version was created using sort -R
  • For some reason, my Splunk instance has NOT indexed these lookup tables, even though they are 15 MB (over the limit)
  • Both files are gzipped
  • I ran the test several times, results were always close this output

araitz
Splunk Employee
Splunk Employee

Lookups are indexed (either in memory or on disk), so I doubt that there is any advantage to presorting the CSV file.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...