Getting Data In

An issue with the HTTP Event Collector (HEC) has been identified in Splunk 7.x

jmaher_splunk
Splunk Employee
Splunk Employee

Summary:
After upgrading from Splunk Enterprise or Splunk Cloud 6.x to 7.x, customers are reporting a bug with HTTP Event Collector (HEC). As a result:

  • Some HEC events may not be getting ingested after the upgrade
  • There may be a reduction in performance (indexing throughput) related to HEC events.

What happened:
Splunk Enterprise and Splunk Cloud releases 7.x (“7.x”) include a limit on HTTP Event Collector (HEC) payloads of 512KB. This limit exists to prevent memory overuse. Post-7.0.x, HEC events with sizes exceeding 512KB are not resolved by the HEC parser, and may be dropped.

Which customers are impacted:
This issue may impact any customer meeting the following criteria:

  1. Are on Splunk Enterprise or Splunk Cloud 7.x
  2. Use HEC
  3. Have a payload size above 512KB
0 Karma
1 Solution

jmaher_splunk
Splunk Employee
Splunk Employee

Resolution:

  • Splunk is working on a resolution to ensure the HEC module in Splunk Enterprise and Splunk Cloud 7.x is more tolerant of larger payloads by default, and we also plan to make the limit configurable to suit specific needs.
  • Splunk Cloud customers that are potentially impacted, will be contacted over the next few weeks to schedule a maintenance window
  • For Splunk Enterprise customers that are potentially impacted, this will be fixed in 7.0.5 (ETA July 27) and 7.1.3 (End of August). We will post to this thread as the maintenance releases are available.

View solution in original post

jmaher_splunk
Splunk Employee
Splunk Employee

Resolution:

  • Splunk is working on a resolution to ensure the HEC module in Splunk Enterprise and Splunk Cloud 7.x is more tolerant of larger payloads by default, and we also plan to make the limit configurable to suit specific needs.
  • Splunk Cloud customers that are potentially impacted, will be contacted over the next few weeks to schedule a maintenance window
  • For Splunk Enterprise customers that are potentially impacted, this will be fixed in 7.0.5 (ETA July 27) and 7.1.3 (End of August). We will post to this thread as the maintenance releases are available.

adammike
New Member

and we also plan to make the limit configurable to suit specific needs

How do I configure this? I can't find anything in the docs or online

0 Karma

sylim_splunk
Splunk Employee
Splunk Employee
maxEventSize = <positive integer>[KB|MB|GB]
* The maximum size of a single HEC (HTTP Event Collector) event.

https://docs.splunk.com/Documentation/Splunk/8.0.5/Admin/Inputsconf 

0 Karma

shankern
Explorer

Is gzip content encoding header supported on HEC ? Would be useful while posting large payloads.

0 Karma

jmaher_splunk
Splunk Employee
Splunk Employee

Update:

The latest maintenance release, 7.0.5, for Splunk Enterprise and Splunk UniversalForwarder are now available from the Download site.
Please note as 7.0.5 is not the latest version, you can find it under the “Older Releases” section.

Download: https://www.splunk.com/en_us/download.html
Known Issues: http://docs.splunk.com/Documentation/Splunk/7.0.5/ReleaseNotes/Knownissues
Fixed Issues: http://docs.splunk.com/Documentation/Splunk/7.0.5/ReleaseNotes/Fixedissues

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...