We are trying to send data to Splunk HEC via Kinesis Firehose but for some reason Firehose keeps logging "Could not connect to the HEC endpoint. Make sure that the HEC endpoint URL is valid and reachable from Kinesis Firehose." We've tried a combination of the following with no luck:
https://hostname.test.com:8088
https://hostname.test.com:8088/services/collector
https://hostname.test.com:8088/services/collector/raw
We are referencing this post: Power Data Ingestion into Splunk which indicates the first https://hostname.test.com:8088 with a raw endpoint should have worked. I'm able to post events via curl using batch and the raw endpoint and json and the event endpoint. This tells me the ELB is working and forwarding events. So I'm wondering what others have set for their Splunk Cluster Endpoint and Splunk endpoint type in Firehose?
Raw Endpoint:
curl -k "https://hostname.test.com:8088/services/collector?channel=00872DC6-AC83-4EDE-8AFE-8413C3825C4C" -H "Authorization: Splunk token" -d '127.0.0.1 - admin [28/Sep/2016:09:05:26.875 -0700] "GET /servicesNS/admin/launcher/data/ui/views?count=-1 HTTP/1.0" 200 126721 - - - 6ms 127.0.0.1 - admin [28/Sep/2016:09:05:26.917 -0700] "GET /servicesNS/admin/launcher/data/ui/nav/default HTTP/1.0" 200 4367 - - - 6ms 127.0.0.1 - admin [28/Sep/2016:09:05:26.941 -0700] "GET /services/apps/local?search=disabled%3Dfalse&count=-1 HTTP/1.0" 200 31930 - - - 4ms
127.0.0.1 - admin [28/Sep/2016:09:05:26.954 -0700] "GET /services/apps/local?search=disabled%3Dfalse&count=-1 HTTP/1.0" 200 31930 - - - 3ms
127.0.0.1 - admin [28/Sep/2016:09:05:26.968 -0700] "GET /servicesNS/admin/launcher/data/ui/views?digest=1&count=-1 HTTP/1.0" 200 58672 - - - 5ms'
Events Endpoint:
curl -k "https://hostname.test.com:8088/services/collector?channel=00872DC6-AC83-4EDE-8AFE-8413C3825C4C" -H 'Authorization: Splunk token' -d '{"event": "Hello"}'
You could use Cribl to pull the data directly from a Kinesis Stream. This has the benefits of avoiding the extra cost of sending data through the Kinesis Firehose + the ability to process the data before sending it to Splunk (or lots of other places)
For Kinesis Firehose, you'll need to have some prerequisites validated prior to sending data into Splunk via Kinesis Firehose.
First, make sure you are using Splunk version 6.6+ . This is required for the HEC health status check. Next, you'll need to have a valid signed SSL certificate on the AWS ELB and a publicly facing IP with sticky sessions enabled. The Splunk Indexers (where the data will be landing from the ELB via HEC) should have the Splunk Add-on for Kinesis Firehose installed and set the stanza ackIdleCleanup = true on the inputs.conf .
Once all that has been done, then you can test your Splunk setup by running the following curl command:
curl https://http-inputs-firehose-<customer>.splunkcloud.com/services/collector/raw?channel=FE0ECFAD-13D5... -H "Authorization: Splunk <HEC_TOKEN>" -d '<raw data string>'
Note that Splunk Cloud does not use the port 8088, but your custom build Splunk instance might.
if the splunk instance is 6.7+, there isn't a need for the channel parameter in the POST