Getting Data In

Fluentd HEC Output: How to target and utilize parts of a tag to configure my index, sourcetype, and host dynamically?

tprz
Explorer

I've got a bunch of custom syslog traffic flowing to a fluentd tier I have running in kubernetes. I'm using the rewrite_tag_filter plugin to set the tag of all the events to their target index. I then use another layer of that plugin to add the host and sourcetype values to the tag.

I'm sending all of that to the same output:

   @type splunk_hec
   index main
   sourcetype ${tag_parts[1]}
   host ${tag_suffix[2]}
   source ${tag}
   hec_host HEC_Host
   hec_port HEC Port
   hec_token HEC Token
   ca_file /fluentd/etc/server.pem

In the configs above I'd like to target different parts of the tag to configure my index, sourcetype, and host dynamically.

The sourcetype and host lines translate those directly to a string, so in Splunk for example I see the host field literally set to "${tag_suffix[2]}"

But the source field I'm setting as a test work and the source field in Splunk contains the whole tag.

How can I target and utilize parts of the tag to configure my settings? Or is there a better way to set these values?
Trying to avoid index time operations on my indexers.

Thanks!

Sources:
I found the prefix, suffix, and parts for tag targeting in record transformer and wasn't sure if they would work
https://docs.fluentd.org/filter/record_transformer

Fluentd to Hec plugin, latest version
https://github.com/splunk/fluent-plugin-splunk-hec

0 Karma
Get Updates on the Splunk Community!

Modern way of developing distributed application using OTel

Recently, I had the opportunity to work on a complex microservice using Spring boot and Quarkus to develop a ...

Enterprise Security Content Update (ESCU) | New Releases

Last month, the Splunk Threat Research Team had 3 releases of new security content via the Enterprise Security ...

Archived Metrics Now Available for APAC and EMEA realms

We’re excited to announce the launch of Archived Metrics in Splunk Infrastructure Monitoring for our customers ...