Simple host field change - does not work

[bjoernhansen]

Hey guys,

so I'm rather new to Splunk, and we're implementing a small cluster for logfile collection and SIEM purposes.
One of our systems is a MobileIron Core Appliance, which features a built-in Splunk forwarder 6.0.3, while the rest of our Splunk system is running 6.5.3.
This system is using a fixed hostname for the events forwarded to our Splunk system, and the system automatically uses the configured (external) system name, but I'd like to use our internal host name for these events.

So, following the documentation and a few threads here on Splunk answers, I tried to set up the following:

/opt/splunk/etc/master-apps/_cluster/local/props.conf

[host::vsp.my-domain.de]
TRANSFORMS-host_rename = host_rename-vsp.my-domain.de

/opt/splunk/etc/master-apps/_cluster/local/transforms.conf

[host_rename-vsp.my-domain.de]
REGEX = .
DEST_KEY = MetaData:Host
FORMAT = host::somehost.local.lan

This all is being distributed by the cluster master through the cluster bundle - I verified the files on the indexers.
Still, all events show up with the original hostname.

I have this in the cluster bundle to:

/opt/splunk/etc/master-apps/_cluster/local/props.conf

[host::10.0.1.2]
TRANSFORMS-drop = drop-loadbalancer

[host::10.0.1.3]
TRANSFORMS-drop = drop-loadbalancer

/opt/splunk/etc/master-apps/_cluster/local/transforms.conf

[drop-loadbalancer]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

These change works perfectly fine, so I'm pretty sure it's not an issue with the cluster distribution or the regex.

Any idea on what kind of stupid mistake I might have made?

[somesoni2]

Does the host name exactly matches vsp.my-domain.de in your events? Do you want to do it for any specific sourcetype OR all sourcetypes?

Also, if the host name is correct, give this a try. I generally prefer more friendly stanza names.

/opt/splunk/etc/master-apps/_cluster/local/props.conf

[host::vsp.my-domain.de]
TRANSFORMS-host_rename = host_rename_vsp_my_domain_de

/opt/splunk/etc/master-apps/_cluster/local/transforms.conf

 [host_rename_vsp_my_domain_de]
 REGEX = .
 DEST_KEY = MetaData:Host
 FORMAT = host::somehost.local.lan

[bjoernhansen]

Yes, the domain matches exactly. I even did a tcpdump to make sure that the event being sent does not include something like a weird UTF-8 character, a space or something like that, but it doesn't. My debug activity shows me that at least the props.conf stanza actually matches:
04-16-2017 22:09:09.085 +0200 DEBUG PropertiesMapConfig - Pattern 'host_rename_vsp' matches with priority 100
And, as you can see - I already changed the stanza name to make sure that no strange behaviour regarding the "-" is the reason for my problems.

[somesoni2]

The host:: requires the full pattern to be matched. If vsp.my-domain.de is just the domain and not the actual host name, then you should add a wildcard in front to create a placeholder for hostname. (use [host::*vsp.my-domain.de] instead of [host::vsp.my-domain.de] in props.conf).

[bjoernhansen]

That's actually the complete hostname. When I turned on debugging, Splunk with debug turned on actually reported that my stanza matched - so that part seems to be okay.

[DalJeanis]

Try adding this to the [host_rename-vsp.my-domain.de] stanza and see if it makes a difference.

SOURCE_KEY=MetaData:Host

All I can guess is that it's looking at some other field which doesn't exist.

[bjoernhansen]

Hey,
thanks for the suggestion. Unfortunately, I already tried that 😉
I must have tried each and any possible combination by now...

[woodcock]

This looks good to me; perhaps your testing assumptions are wrong. Indexed data is immutable. Therefore configuration changes to index-time settings (like this one) will only effect data that is indexed after the configuration changes are deployed AND after the splunk instances on the Indexers (or Heavy Forwarders) are restarted (old events will stay wrong).

[bjoernhansen]

I know this will only affect new events, but that UF is pushing events almost continously, so it's easy to monitor for it. I tried manually restarting the indexers. The one props-transform (see above) works fine - the other is still broken. I have no clue...

[bjoernhansen]

As it seems I can not edit my question, here is a pastebin of the btool output for props and transforms regarding my stanzas:
https://pastebin.com/GhjyqJan

I also tried to rename the stanza to host_rename_vsp to avoid any issues with "-", but to no success.

[bjoernhansen]

I restarted both indexers with --debug for half a minute.
The log actually states that the props stanza matches:

04-16-2017 22:09:09.085 +0200 DEBUG PropertiesMapConfig - Pattern 'host_rename_vsp' matches with priority 100

Nevertheless - no change at all.