I've seen a lot of discussion and recommendations on the number of file descriptors that should be made available to a Splunk instance, but should that vary depending on the primary function of the instance? Does a Search Head require more than a Universal Forwarder for instance?
Similarly for threads, which I assume should be higher for a busy Search Head or Indexer, but not so high for a Deployment Server or License Master - is that correct?
Is there any way to look at a running instance and determine how much of these resources are in use?
Should I have a standard base configuration for all instances or should it vary depending on the function/overall workload?
You can install the splunk on splunk app. This app has a feature which allows you to view file descriptor usage across multiple instances of splunk. As a general best practice you should try to have all instances of splunk have at least 8192 open file descriptors. Two good posts I often refer people to are the following
http://blogs.splunk.com/2011/11/21/whats-your-ulimit/
http://answers.splunk.com/answers/13313/how-to-tune-ulimit-on-my-server