Deployment Architecture

Increasing Splunks Search performance on Linux

dragmore
Explorer

Hi. We have several big Splunk installations and im working on trying to increase the search performance on them. Unfortunatly ive come to and end and i could really use some input/suggestions on where to fix this.

Info:
1. Splunk 4.3.2 x64 REDHAT @ RHEL 5.7 X64
2. HOT/WARM IDX @ 2x120GB SSD in RAID1 mounted volume
3. COLD & Thawed @a 14x300GB RAID6-ADG mounted volume
4. 2x6CPU Cores and 48GB MEM (HP DL380g7)

So when i do a search i often see almost all my cpu's ad idle, but the one im using for search..
I got no IO-Waiting on my Disk-IO subsystem so i know this issue is CPU bound.

So the BIG question is : Is there a way to enable a search to span over multiple cpu cores? Multithreaded/processed searches?

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 196 812064 1117692 32270052 0 0 6 50 2 2 6 0 94 0 0
2 0 196 809468 1117708 32272140 0 0 0 257 1280 1743 9 0 90 0 0
7 0 196 660532 1117800 32270748 0 0 62 1726 1602 3894 25 2 72 0 0
7 0 196 556972 1117920 32274096 0 0 1 1690 1648 21236 50 1 48 0 0
3 0 196 687980 1117952 32258168 0 0 0 428 1424 10324 40 1 59 0 0

br TE

0 Karma

twkan
Splunk Employee
Splunk Employee

Personally, I would install multiple Splunk Indexers listening on different ports with the aim of saturating the CPU cores as well as Disk I/O. Given that you have 12 CPU cores, I would start with perhaps 2 to 3 Splunk instances, and monitor the health status via iostat, top etc. to make sure that I am not overloading the box, and subsequently validate the improved utilisation of the hardware resources.

MuS
SplunkTrust
SplunkTrust

okay at first I disagreed on this but after reading http://splunk-base.splunk.com/answers/5202/how-do-i-get-the-most-out-of-a-16-core-server I think you can improve search performance this way.

0 Karma

twkan
Splunk Employee
Splunk Employee

Generally speaking, search performance will increase along with indexing performance. This is where the multiple indexers with MapReduce will come into play to increase the search performance.

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi twkan, then you would 'only' increase the index performance but not the search performance.

0 Karma

MuS
SplunkTrust
SplunkTrust

Hi dragmore

please read this answer to find out more about search performance.

regards

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...