I would like to understand better how transformations work, in terms of priority and data flow.
Let's say I have 3 transformations in place, that I want to apply to a sourcetype (e.g. "test"):
[tr1]
[tr2]
[tr3]
Can somebody explain to me what the difference is (if any) between these 2 props.conf:
----1----
[test]
TRANSFORMS-1=tr1
TRANSFORMS-2=tr2
TRANSFORMS-3=tr3
---2----
[test]
TRANSFORMS-1=tr1,tr2,tr3
Is the resulting data transformation the same on both cases?
Thank you!
As I understand it, transforms listed separately are processed in lexicographical order whereas those listed in a single TRANSFORMS setting are processed in the order given.
IOW, this
[test]
TRANSFORMS-3=tr3
TRANSFORMS-1=tr1
TRANSFORMS-2=tr2
is the same as this
[test]
TRANSFORMS-1=tr1
TRANSFORMS-2=tr2
TRANSFORMS-3=tr3
But this
TRANSFORMS-1=tr3,tr1,tr2
will process the transforms in the listed order.
So, if they are processed in lexicographical order, then the result should be the same once the data passes through my 2 transformation examples.
Best practice, as I understand it, is to list the transformations in the second form
TRANFORMS=tr1,tr2,tr3
so that there is no doubt on the order they are processed.
hi @cmlombardo ,
the order of transformation is relevant!
only for example, if you read at https://docs.splunk.com/Documentation/Splunk/9.1.1/Forwarding/Routeandfilterdatad#Filter_event_data_... , if you want to keep some events and discard the rest, you have to executebefore the transofrmation on all the events (regex=.) and then the tranformation of a part of data, if you change the order, the tranformation doesn't work.
Ciao.
Giuseppe
It's not necessarily true that the data will be the same after each example. In some (many/most?) cases, the order of transformations could be significant and is why I recommend using the second format.
In the example, the lexicographic order will process the transformations in the same order, as TRANSFORMS-1 comes before TRANSFORMS-2 (and so on).
Hi @cmlombardo,
I'm not sure (someone else could contradict me!) but they should be the same thing.
Anyway, I prefer the second solution to be more sure about the sequence of transformation application.
because the execution sequence could be important.
Ciao.
Giuseppe