All Apps and Add-ons

How come my AWS Kinesis Firehose is failing to connect to HEC due to SSL Handshake?

gf13579
Communicator

Has anyone successfully achieved Kinesis Firehose to a HEC secured with letsencrypt certs?

I've used letsencrypt to generate SSL certs for my Splunk server. I've used those in web.conf to secure Splunk web, and I'm trying to use them with HEC to permit SSL connections.

I've got as far as cURL submitting events using HTTPS (without -k to ignore cert errors!) and if I browse to https://splunk.mydomain.com:8088, my browser is happy with the cert.

The problem is that AWS Kinesis Firehose isn't happy.

Cloudwatch is reporting

{
    "deliveryStreamARN": "arn:aws:firehose:us-west-2:123455522430:deliverystream/my-delivery-stream",
    "destination": "https://splunk.mydomain.com:8088",
    "deliveryStreamVersionId": 1,
    "message": "Could not connect to the HEC endpoint. Make sure that the certificate and the host are valid.",
    "errorCode": "Splunk.SSLHandshake"
}

web.conf looks like this:

[settings]
enableSplunkWebSSL = true
privKeyPath = etc/auth/mydomain.com/privkey.pem
caCertPath = etc/auth/mydomain.com/fullchain.pem

/opt/splunk/etc/apps/splunk_httpinput/local/inputs.conf currently looks like this

[http]
disabled = 0
index = temp
enableSSL = 1
sslVersions = *,-ssl2
allowSslCompression = true
allowSslRenegotiation = true
caCertFile =  /opt/splunk/etc/auth/mydomain.com/cert.pem
sslKeysfile = /opt/splunk/etc/auth/mydomain.com/privkey.pem
sslKeysfilePassword =
ackIdleCleanup = true

I've tried using openssl to produce a password-protected key file (privkey.pem) and specify sslKeysfilePassword — no difference.

I read one answers post saying that letsencrypt wasn't trusted by AWS, but that seemed unlikely. All help appreciated!

I'm using Splunk Enterprise 7.2.4.

1 Solution

gf13579
Communicator

Similar to other threads on this, I managed to get Kinesis Firehose -> HEC working by doing the following:

  • Standup a splunk server in AWS
  • Configure it to listen with HEC and disable HTTPS
  • Put an ELB in front of it, listening for HTTPS on 8088 and forwarding to HTTP (not S) on 8088
  • Create a DNS CNAME record for splunk.mydomain.com -> myloadbalancerid.elb.amazonaws.com
  • Use AWS ACM to issue a cert for that name and associate it with the ELB
  • Create a Firehose data stream sending data to https://splunk.mydomain.com:8088

It's frustrating to not know why Firehose wasn't happy sending to my original HEC - potentially due to LetsEncrypt being the CA but that's just speculation.

View solution in original post

bullet
Explorer

I know that this is an old thread, however, after a lot of browsing I was able to make AWS pipe AWS Flow logs to Splunk HEC with letsencrypt cert and without needing to use ACM.

 

All one has to do is see if they get correct letsencrypt cert when you browse to HEC endpoint like so.

curl -k https://yoursplunkinstanceDOTcom:8088/services/collector/event -H "Authorization: Splunk <Token>" -d '{"event":"hello world"}' -v

If you don't see your letsencrypt cert here.  You have to create a combinedsplunk.pem from these three files.

cd /etc/letsencrypt/live/your-server-hostname/
cat cert.pem privkey.pem chain.pem > /opt/splunk/etc/auth/viewdns/combinedsplunk.pem
chmod 600 /opt/splunk/etc/auth/viewdns/combinedsplunk.pem
chgrp splunk:splunk /opt/splunk/etc/auth/viewdns/combinedsplunk.pem

 

Then you have to create inputs.conf file and ensure it is readable by splunk user.

cd /opt/splunk/etc/system/local/

cat inputs.conf
[http]
disabled = 0
index = main
enableSSL = 1
serverCert = /opt/splunk/etc/auth/yoursplunkinstance/combinedsplunk.pem
sslPassword =
crossOriginSharingPolicy = *

 

Then restart splunk.

/opt/splunk/bin/splunk restart

 

Now you should be able to go to HEC without any SSL error even without the -k switch in curl which asks curl to ignore SSL cert errors.

curl https://yoursplunkinstanceDOTcom:8088/services/collector/event -H "Authorization: Splunk <Token>" -d '{"event":"hello world"}' -v

gf13579
Communicator

Similar to other threads on this, I managed to get Kinesis Firehose -> HEC working by doing the following:

  • Standup a splunk server in AWS
  • Configure it to listen with HEC and disable HTTPS
  • Put an ELB in front of it, listening for HTTPS on 8088 and forwarding to HTTP (not S) on 8088
  • Create a DNS CNAME record for splunk.mydomain.com -> myloadbalancerid.elb.amazonaws.com
  • Use AWS ACM to issue a cert for that name and associate it with the ELB
  • Create a Firehose data stream sending data to https://splunk.mydomain.com:8088

It's frustrating to not know why Firehose wasn't happy sending to my original HEC - potentially due to LetsEncrypt being the CA but that's just speculation.

TheKennyD
Observer

Did you ever figure out how to make the HEC use a trusted SSL certificate instead of the default self-signed cert? 

0 Karma

obla
New Member

Hey @gf13579 did you setup an internal or external elb for this?
Also what did you add to your elb security group?

0 Karma

gf13579
Communicator

Hi @obla. I setup an external (scheme: internet-facing) one though, assuming you can issue certs from ACM to internal ELBs, internal should be fine.

The ELB's SG contains inbound rules for TCP 8088 and 443 from 0.0.0.0/0 though it should just need 8088 or whatever port you configure FireHose to send on. How you restrict inbound traffic to the FireHose is beyond me at this point.

Outbound is unrestricted but could've been limited to 8088 to the security group of the EC2 VM hosting splunk.

Not a very locked-down setup but it was for a temporary(...) solution.

0 Karma

TheKennyD
Observer

Hello, 

Is there really no way to get Splunk's HEC to answer SSL requests using a trusted CA certificate?

I used to use a solution to send AWS Security Hub data to Splunk's HEC via a HTTP. Then along came TRUMPET which requires a trusted SSL certificate and will not used Splunk's self signed certificate. So I went out and purchased a self signed cert, then configured the Splunk web server to use it. However, the HEC is still using Splunk's self signed certificate. I've looked at several attempts online to configure server.conf and input.conf to point to that certificate but none of those have worked. 

Does anyone know how to configure Splunk's HEC to use SSL configured with a trusted certificate? 

Thank you, 

Ken 

 

 

 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...