@Gerrykahn:
To allow a VPC-enabled Lambda function access to public internet, you need to attach it to a private subnet with internet access though a NAT instance or a VPC NAT Gateway. Create this new private subnet in the same VPC where you Splunk instances live to give Lambda function connectivity to both:
KMS service: via public internet (through NAT instance/gateway)
Splunk servers: via local route
A side benefit of a dedicated private subnet is that you'll have a larger pool of private IP addresses available to Lambda to scale as it sets up network interfaces for your Lambda functions.
With respect to encrypting the Splunk HEC token: with Lambda support of environment variables (released just days after this question was posted), Lambda automatically encrypts these variables by default (using KMS), and decrypt them for you per function invocation. That may be sufficient depending on your security requirements. You can also manually encrypt the environments variables before you deploy Lambda, as you have previously done, where Lambda function code will have to decrypt them at run-time and will therefore need access to KMS.
By the way, Splunk has released several basic Lambda blueprints to collect data from different AWS services such as DynamoDB, Kinesis and CloudWatch Logs including a generic logging one. There's also a step-by-step walkthrough on how to set them up. Thought you may be interested in those as well.
... View more