It seems that the transformation layer only processes an event once. If the factors that influence which props.conf stanza are applied, this does not cause the event to continue to be processed. Is there any mechanism to make this happen?
The usecase is for a file that has multiple event types inside. I first determine the sourcetype as a series of transforms and then would like to set the properties of that sourcetype in props.conf.
I have tried using a heavy forwarder in front to do the initial sourcetype decision making but the indexer seems not to apply any transformations to forwarded events. Is there a way to make this happen?
Thanks, Tom
It depends on how/where you change it and what you settings you are trying to invoke (which parser handles them). If you "change" it with rename
, which is a search-time
operation, then definitely not do anything at all. The Splunk_TA_paloalto
TA from SplunkBase take stuff that comes in with sourcetype=pan:logs
and breaks it out (successfully) like this:
From props.conf:
[pan:log]
TRANSFORMS-sourcetype = pan_threat, pan_traffic, pan_system, pan_config, pan_hipmatch, pan_endpoint
From transforms.conf:
[pan_threat]
DEST_KEY = MetaData:Sourcetype
REGEX = ^[^,]+,[^,]+,[^,]+,THREAT,
FORMAT = sourcetype::pan:threat
And then later in props.conf for some stuff:
[pan:threat]
TIME_PREFIX = ...
REPORT-foo...
FIELDALIAS-foo ...
EVAL-foo ...
LOOKUP-foo ...
The bottom line, though, is that you are never going to get a second change to go through the TRANSFORMS-
parser unless you cook
it twice (Index it twice). Once it has been cooked, the only decision to be made is where to write it to disk. Can you transform it with syslog-ng
before sending it to Splunk (this is a very common way to handle this kind of thing)? Another possibility might be manipulation with HTTP Event Collector
; as I recall, it is very special in how it cooks
the data.