All Apps and Add-ons

HEC integration with AWS Kinesis Firehose

sundaysky
Engager

Hi,

We're trying to send data to a our single Heavy Forwarder via the new destination type of Splunk in Kinesis Firehose. We've configured the Firehose to send data as "event" type (rather than "raw") since we use a transformation AWS Lambda to configure the http requests body as JSON (as described in Format HEC events documentation) but for some reason, Firehose fails with reason:

The connection to Splunk timed out. This might be a transient error and the request will be retried. Kinesis Firehose backs up the data to Amazon S3 if all retries fail
But when I try to curl to the HEC endpoint from my laptop, or any other network, with the relevant token, I see messages in Splunk and flow is valid.

Any suggestions? Where at least can I see logging for HEC to verify that requests arrive to Splunk and see their error message? (and not just metrics...)

I'm using version Splunk Enterprise 6.5.1 with AWS Kinesis Firehose Add-on for Splunk
Thanks.

0 Karma

eprokop_splunk
Splunk Employee
Splunk Employee

Hi there,

AWS Kinesis Firehose to Splunk requires Splunk version 6.6 or higher.

For future reference, there is an HEC dashboard in the Monitoring Console that shows a summary of activity and errors. In Splunk Enterprise, it's found under Settings -> Monitoring Console -> Indexing -> Inputs -> HTTP Event Collector.

HEC logs are in the splunkd logs found in index=_internal (filter by source for the splunkd logs on disk). Some keywords to search for include HttpEventCollector, HttpInputDataHandler, HttpInputEventParser, HttpInputQueue, and HttpInputServer. You can enable different logging levels for these components under Settings -> Server Settings -> Server Logging.

Hope this helps!

Eric

Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...