I have an issue where we have a sourcetype that we want to remove a transform (on the indexer) that drops some data (but only for a specific host wildcard pattern). I've been tasked with keeping the same sourcetype name, but not applying the transforms to outputs from a series of hosts.
We already use a props [Host: ] stanza for timezones. Is there a simple way to keep the same sourcetype, but force inputs from a particular host pattern to follow a different set of transforms? Preferrably a host (and sourcetype) specific transform.
Thanks
It can be hard to perform both a host and sourcetype-specific transformation. It depends on what you want to be done.
In props.conf, select the sourcetype. First, apply the usual transformation, then apply the exception case
[mysourcetype]
TRANSFORMS-t1=firstTransform,secondTransform
In transforms.conf, send the data as needed. In the first transform, do whatever you are doing now. I just put in the nullQueue to have a complete example. The second transform selects the data based on the host name, not based on the event content.
[firstTransform]
REGEX = what_to_delete_based_on_raw_data
DEST_KEY = queue
FORMAT = nullQueue
[secondTransform]
SOURCE_KEY = MetaData:Host
REGEX = pattern_to_match_host_name
DEST_KEY = queue
FORMAT = indexQueue
You can read more about this at Filter Event Data and Send to Queues
Please test this carefully and be sure to use regular expressions, not just "wildcards." I can't test it, so I might have made mistakes...
It can be hard to perform both a host and sourcetype-specific transformation. It depends on what you want to be done.
In props.conf, select the sourcetype. First, apply the usual transformation, then apply the exception case
[mysourcetype]
TRANSFORMS-t1=firstTransform,secondTransform
In transforms.conf, send the data as needed. In the first transform, do whatever you are doing now. I just put in the nullQueue to have a complete example. The second transform selects the data based on the host name, not based on the event content.
[firstTransform]
REGEX = what_to_delete_based_on_raw_data
DEST_KEY = queue
FORMAT = nullQueue
[secondTransform]
SOURCE_KEY = MetaData:Host
REGEX = pattern_to_match_host_name
DEST_KEY = queue
FORMAT = indexQueue
You can read more about this at Filter Event Data and Send to Queues
Please test this carefully and be sure to use regular expressions, not just "wildcards." I can't test it, so I might have made mistakes...
This is a really good answer. Thank you.
The props/transforms I'm working on have a mix of REPORT / EXTRACTS. I endedup adding the rule at the end of the REPORT section and this looks to have done the trick in our development instance.