Getting Data In

Fluentd HEC Output: How to target and utilize parts of a tag to configure my index, sourcetype, and host dynamically?

tprz
Explorer

I've got a bunch of custom syslog traffic flowing to a fluentd tier I have running in kubernetes. I'm using the rewrite_tag_filter plugin to set the tag of all the events to their target index. I then use another layer of that plugin to add the host and sourcetype values to the tag.

I'm sending all of that to the same output:

   @type splunk_hec
   index main
   sourcetype ${tag_parts[1]}
   host ${tag_suffix[2]}
   source ${tag}
   hec_host HEC_Host
   hec_port HEC Port
   hec_token HEC Token
   ca_file /fluentd/etc/server.pem

In the configs above I'd like to target different parts of the tag to configure my index, sourcetype, and host dynamically.

The sourcetype and host lines translate those directly to a string, so in Splunk for example I see the host field literally set to "${tag_suffix[2]}"

But the source field I'm setting as a test work and the source field in Splunk contains the whole tag.

How can I target and utilize parts of the tag to configure my settings? Or is there a better way to set these values?
Trying to avoid index time operations on my indexers.

Thanks!

Sources:
I found the prefix, suffix, and parts for tag targeting in record transformer and wasn't sure if they would work
https://docs.fluentd.org/filter/record_transformer

Fluentd to Hec plugin, latest version
https://github.com/splunk/fluent-plugin-splunk-hec

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...