Splunk Search

Why kv store lookup is much slower than csv lookup?

iKate
Builder

Hi everyone!
We've moved some of heavy lookups to kv store and now they work faster and more stable. But one of them barely can calculate something, being inserted in a search with lookup even on 24 hour span, while it easily loads with inputlookup.

Accelerated field is a hexadecimal uid of 16 symbols. To make it easier for splunk to accelerate it, we converted it in a bigint and accelerated on this new field. Still the same slowness. It was rewritten but it didn't help. And its csv version lookups much faster, but we had some issues with updating large lookups so we moved to kv.

Slow-lookuped kv has 10mln entries and around 5 fields.
Fast-lookuped kv with 40mln entries and 10 fields works fine (with accelerated field also converted from hex to bigint)

Server: 251.81 GB Physical Memory, 20 CPU Cores

in limits.conf:
max_threads_per_outputlookup = 10

in server.conf:
oplogSize = 2000

What can cause such slowness? What params can we tune?

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...