@jnicholsenernoc and all - here is my working solution. we use CloudFormation and ansible to deploy everything pretty nicely in about 5-10 minutes. i've had mixed results trying to use Splunk's CFN and ansible tasks for several reasons - so i've rolled my own.
i'm a hands on guy - so here's a code snippet for your reference:
https://bitbucket.org/snippets/asecurityteam/58G8X
there are a few caveats worth explaining though about our setup:
we build our search heads and ELBs using CFN - meaning we don't point and click anything in the AWS Console
the certificate itself is installed already, so in the CFN snippet { "Ref" : "SplunkSearchLoadBalancerSSLCert" } refers to the cert's ARN
the search head instances themselves run SSL on port 8443 so that we don't need to deploy or maintain certs, and also we don't need to run splunk as a privileged user (to use ports < 1024 in linux; you could use iptables or apache or something to redirect on 443 but meh?)
nodes themselves use self-signed certs because it's easier to manage / irrelevant for the most part, and besides the trusted cert sits on the ELB
nodes themselves are in an Auto Scale Group - cause it's easier to build new ones and tie in easily to the ELB all at once... but that shouldn't be a requirement either way
the reason why we're not overly concerned about what port to run or what cert to use on the individual nodes is that we use security groups to control access to the nodes versus the ELBs. you literally can't log in to the search heads individually - only access via the ELB. this line controls the ELB access, and another one elsewhere in our CFN stack does a similar thing for the Search Head cluster members "SecurityGroups" : [ { "Ref" : "SplunkSearchLoadBalancerSG" } ]
i'm not 100% sure this is the best / right LB policy (ie caching, timeouts, retries, etc), so there could be better ways
... View more