Does the Universal Forwarder support the Splunk header

[Marinus]

I recently moved to the universal forwarder (4.3.3) where I collect files using the batch input. It's a long story but I have to use the batch input. I use the SPLUNK header to set the host, source and source type. The receiving indexer performs the necessary transformations. I've noticed since I've moved to the new forwarder that the header is no longer being honoured. I changed the HEADER_MODE to always in the default etc/system/local/props.conf, however events arrive without the necessary host, source and sourcetypes.

It appears that the batch handling changed in 4.2, and is not handled as part of the indexing pipeline.

[Marinus · Accepted solution]

It appears that the batch handling changed in 4.2, and is not handled as part of the indexing pipeline. Reverted back to an old forwarder.

[dart]

Hi Marinus,
I'm not sure if using the ***SPLUNK*** style is supported.
I'd suggest either using the Splunk Forwarder instead of the universal forwarder, or you could set a sourcetype in your batch input, and reference that sourcetype in the TRANSFORMS, which could fix host, source and sourcetype, and also use a SEDCMD to remove the header.

I'd say the better solution is to use a full forwarder, if that works for ***SPLUNK*** style.

dart

[Marinus]

Hi Dart

I did a couple of tests and it doesn't appear that HEADER_MODE config affects they way it processes events 😞

[dart]

What's the sourcetype of your data?

Do you have any transforms of the data? What kind of stanza specification are you using on the indexer for these? [my_sourcetype] or [source::/path/to/file] or [host::host1]?

What are you setting on the forwarder inputs?

[Marinus]

Thanks for the response Dart. The indexer uses a batch input to collect data.

[batch:///data]
move_policy=sinkhole
crcSalt=

The host, source and sourcetype are set by the splunk header i.e.
SPLUNK host=acme source=xyz sourcetype=abc

The indexer received the events from the forwarder and has props configured to deal with the source types, which in fact rewrite the source and host keys i.e.

[abc]
TRANSFORMS-fix=fix_a, fix_b

When I look at the events on the indexer, I can see that raw events including the SPLUNK header, with no keys set.