Scaling kv store performance

[Lucas_K]

I am encountering an issue with the kvstore (6.4.1/6.4.2) where i am hitting a relative performance limit with update/insert records.

I have tried different search head architectures, both standalone and clustered (8 members). Hardward based, vm based, and even an i7 laptop.
I have tried splitting my searches into small timeframes (1 minute per search) so there are less events to update per search.
I have tried staggering my searches so that the small timeframes have even less concurrent kvstore operations occuring at the same time.

Regardless of what I try I hit a limit of between 1500-1800 record updates per minute.

I have a use case where I need to update around 2 million records every 10 minutes. At a 1.5-1.8k update rate I am looking at an 18-22 minute run time for a 10 min window. Due to the cumulative performance limit of the kvstore it doesn't help when I split the searches so more can run concurrently. All it does it slow down the kvstore writes.

Using a search head cluster with multiple kv stores doesn't help as all the writes are delegated to the captain anyway.

I've tried the following limits.conf settings but performance doesn't really change much.

[kvstore]
max_queries_per_batch =  20000
max_rows_per_query = 1000000
max_queries_per_batch = 20000
max_size_per_result_mb = 5000
max_accelerations_per_collection = 0 
max_fields_per_acceleration = 0
max_threads_per_outputlookup = 0

Any suggestions on speeding up the kvstore?

[Lucas_K]

I think it may be to do with mongodb internal indexing of fields.

I'm making an assumption that splunks kv store "accelerated field" is actually a mongodb index or something similar.

Just found this post in regards to kv "indexes" -> https://answers.splunk.com/answers/246404/how-to-create-an-index-on-a-kvstore.html

So it seems that by having my key accelerated im actually slowing down inserts.

This mongodb post seems to suggest that true mongodb indexes actually increase upsert performance. This contradicts the splunk answers post in relation to accelerations.

[masonmorales]

Have you tried setting replicate=true in your collections.conf and local=false in your lookup command? This will allow the lookup to run on the indexers. See http://dev.splunk.com/view/SP-CAAAEZJ#replication

[Lucas_K]

For the size of my kvstore it actually slows it down even more.

The writes first need to be performed by the captain in the search head cluster and then replicated out to over 30 indexers.