This is really more of a discussion topic than an actual question. Definitely something more suited to the IRC channel!
That said however, I will provide some URL's to Docu/Wiki pages that help to build a bigger picture. It should be stressed that deployments can vary massively and its quite hard to cover all bases before going in. Not to mention if you are after system specific issues it needs to be remembered that Splunk operates across alot of different network / hardware configurations and it's unlikely that you'll get a coherent response that is useful to yourself.
Different parts of a deployment;
http://docs.splunk.com/Documentation/Splunk/latest/Installation/ComponentsofaSplunkdeployment
Some best practices and good links to other resources;
http://www.splunk.com/wiki/Deploy
How much space do you need?
http://docs.splunk.com/Documentation/Splunk/latest/Installation/HowHowmuchspaceyouwillneed
Following on from above, Licensing;
http://docs.splunk.com/Documentation/Splunk/latest/Admin/HowSplunklicensingworks
Again from above, hardware capacity;
http://docs.splunk.com/Documentation/Splunk/latest/Installation/CapacityplanningforalargerSplunkdepl...
Considerations for deployment data inputs;
http://www.splunk.com/wiki/Community:Deployment_Considerations
Backup Policy;
http://docs.splunk.com/Documentation/Splunk/latest/Admin/Whatyoucanbackup
Data retention policy;
http://docs.splunk.com/Documentation/Splunk/latest/Admin/Setaretirementandarchivingpolicy
Do you need a hardened install? (Security requirements)
http://www.splunk.com/wiki/Community:DeployHardenedSplunk
That said, the floor is obviously open to anyone with some good issues 🙂
