I have indexing about 1GB data per day, but I have a lot of scheduler searchers. There are about 200 searches that runs every minute. Currently I have two indexers (8CPU and 24CPU) and one search head (24CPU). I noticed that search head is running slower and slower. Splunk instance on search head crashes few times a day and it must be restarted. I need to enlarge my architecture but I don't know in what direction should I go. Do you have any ideas?
there's a chapter in the Distributed Deployment Manual about how Splunk uses different types of resources, here's the topic about search performance:
http://docs.splunk.com/Documentation/Splunk/5.0.2/Deploy/Accommodatemanysimultaneoussearches