Getting Data In

Host override issues

wwwdrich
Explorer

I am trying to override the host field based on an element in the source path. This is data that is being forwarded from a universal forwarder and we want the host the data originally came from as the host field, not the hostname of the forwarder.

To date, I have tried modifying etc/system/local/transforms.conf and props.conf both on the indexers and in the search app on our search heads; all with no luck. I have also tried various values in transforms.conf ranging from both MetaData:Source and source for SOURCE_KEY and using source:: in the regex. Anyone have any ideas what I'm doing wrong?

Here is what is currently in transforms.conf and props.conf:
transforms.conf:

[set-host-elasticsearch]
SOURCE_KEY = MetaData:Source
REGEX = /hosts/([^/]+)/logs/
DEST_KEY = MetaData:Host
FORMAT = host::$1

props.conf:

[elasticsearch]
TRANSFORMS-set-host-elasticsearch = set-host-elasticsearch

The regex works fine in a search, I can run the following and get a table of hosts:

sourcetype=elasticsearch | rex field=source "/hosts/(?<hostname>[^/]+)/logs/" | stats count by hostname

As an example, my source path might be something like:
/rel/ps/applications/elasticsearch/hosts/tvrap571/logs/test_cre_gld.log

Tags (2)
1 Solution

lguinn2
Legend

What is currently in your props.conf and transforms.conf should work - these files should be on all of your indexers.

However, you could also do this more easily on the forwarder using inputs.conf. This solution is also much more efficient.

[monitor://yourinputhere]
host_regex = /hosts/(?<hostname>[^/]+)/logs/

More info here

View solution in original post

lguinn2
Legend

What is currently in your props.conf and transforms.conf should work - these files should be on all of your indexers.

However, you could also do this more easily on the forwarder using inputs.conf. This solution is also much more efficient.

[monitor://yourinputhere]
host_regex = /hosts/(?<hostname>[^/]+)/logs/

More info here

lguinn2
Legend

Here's the deal - a Universal Forwarder (UF) does not do parsing

However, if you assign a sourcetype or host to the entire data stream in inputs.conf, that works. Because the UF doesn't have to parse anything to do it. You can even set values in props.conf on the UF - as long as you are referring to properties that apply at input time and require no parsing. For example, you can override the automatic sourcetype of a source in props.conf on the UF.

0 Karma

wwwdrich
Explorer

Great news, that works!! Thanks for the tip, I didn't think it would honor that on the forwarders.

0 Karma

wwwdrich
Explorer

These are universal forwarders, I thought they didn't support any manipulation of the data? I'll have to give that a try, since the forwarder configs are managed via. the deployment server it's an easy fix. I'll add a comment with how it goes.

Thanks!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...