Hi @DalJeanis
Thank you for your interesting answer, and sorry for my late feedback (it was holiday in France yesterday)
I realised by reading your solution that even it's really a smart one it doesn't provide the desired result because I wasn't clear in my explanation.
Allow me to add an example to illustrate what I want to achieve
Imagine the following event data
001EAAABBBCCCXXX
002EDDDEEEFFFYYY
003RGGGHHHKKKZZZ
Then, with extracting rules from props.conf I will have output fields of my event like this
code = 001,type = E, field1 = AAA, field2 = BBB, field3 = CCC, field 4 = XXX
code = 002,type = E, field1 = DDD, field2 = EEE, field3 = FFF, field 4 = YYY
code = 003,type = R, field1 = GGG, field2 = HHH, field3 = KKK, field 4 = ZZZ
and so on...
The desired result would be something like this
code = 001,type = E, field1 = AAA, field2 = BBB, field3 = CCC, field4 = XXX, unique_key=AAA
code = 002,type = E, field1 = DDD, field2 = EEE, field3 = FFF, field4 = YYY, unique_key=DDDEEE
code = 003,type = R, field1 = GGG, field2 = HHH, field3 = KKK, field4 = ZZZ, unique_key=GGGKKKZZZ
...etc
This is what I meant by "applying extracting rule defined in the lookup table"
code, type, key_fields
001, E, field1
002, E, field1 + field2
003, R, field1 + field3 + field4
Some additional details if it could help
- there is approx. 50 fields in total, but the ones which is potentially needed to form the "unique_key" don't exceed 10 (your first question above)
- the order has to be exactly the same as configured in props.conf (or in something else) (your 2nd question)
- values from the fields used to created "unique_key" could not be null or blank
... View more