How many search heads should I have in my environment for X amount of users?

[MikeBertelsen]

I’m at the point where I think I need to increase the number of search heads for the current usage base and future user growth.

Currently there is just the one search head serviced by two load balanced indexers.
I’m thinking one additional SH for sure, but maybe more as Splunk continues to increase in popularity.
I’m also planning to add two additional indexers which will be load balanced with the existing two.
The current average for indexing is about 300 g per day. This too is expected to grow.

Over the past 90 days, one SH handled about 125 users. The SH does not appear to be stressed, and most users are low volume searchers. I expect the number of users to grow to about 400 by this time next year.

The search I used is slightly adjusted from Splunk’s user activity search:

index=_audit action=search (id=* OR search_id=*)  | eval search_id = if(isnull(search_id), id, search_id)  | replace '*' with * in search_id  | search search_id!=rt_*  | rex "search='(?<search>.*?)', autojoin"  | search search_id!=scheduler_*  | convert num(total_run_time)  | eval user = if(user="n/a", null(), user)  | stats min(_time) as _time first(user) as user max(total_run_time) as total_run_time first(search) as search by search_id  | search search=search* search!=*_internal* search!=*_audit* | top limit=300 user

Attached are the results of the search.

Suggestions please.

[masonmorales · Accepted solution]

There is no hard and fast rule on this because there are too many variables. You may have X number of users but only Y are performing searches at any given time, with Z types of searches, and so on.

The general rule is to always add indexers before search heads. Adding indexers allows search results to return quicker for all users. You should add a search head if you are frequently hitting the concurrent search limit and your indexers are not being overtaxed.

If you just want to be prepared for having 400+ users at the search layer, deploy 3 search heads, cluster them, and put an external load balancer in front of them. Then, just keep adding indexers as your daily ingest volume grows.

You should also check-out the capacity planning docs if you haven't already done so: http://docs.splunk.com/Documentation/Splunk/6.3.0/Capacity/IntroductiontocapacityplanningforSplunkEn...