Monitoring Splunk

Reasonable Search performance?

lee28
New Member

Hi,
We ran a search command(just count the total event) and got the following results. (using 3 indexers)
total event count = 82,843,934

duration = 2,413.578 sec

Is it reasonable? looks to me that the search speed is quite slow.
Is there any way to increase the search performance?
Changing the settings in 'limits.conf or 'times.conf' file will help?

Thanks in advance
Julian

Tags (1)
0 Karma

dwaddle
SplunkTrust
SplunkTrust

Well, "reasonable" is sometimes subjective. Just doing some basic maths here -- 82,843,934 events / 2,413 secs = 34,332 events per second scan rate. If you divide that by the number of indexers (assuming the data is perfectly distributed, which may not be true) that is 11,444 events per second per indexer.

Another assumption that each event is 1000 bytes (which may not be true) puts your throughput around 11 MBytes/sec - which is low relative to the basic throughput of a modern disk subsystem. You do have an appropriate disk subsystem attached, right? And these are physical machines, or VMs?

This also includes overhead from search-head to indexer coordination, CPU-time cost of doing field extraction, and a few other things. You really don't have the information to see where all the time was spent. There's a search job inspector tool that can help. Perhaps you can update with data from it?

But, I think there is a bit of misconception here. A search to "count ALL the things!" is not really a objective test of search performance. You need to search for something other than "everything". A highly dense search (where the number of events returned is a large fraction of the total number of events in the system) will usually be slower than a relatively sparse one.

Get Updates on the Splunk Community!

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...

Introducing Splunk Enterprise 9.2

WATCH HERE! Watch this Tech Talk to learn about the latest features and enhancements shipped in the new Splunk ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...