Starting with the data in an event:
Lines in Single Event:
PosTransactionProperties[1].PosTransactionPropertyCode[1]=RECEIPT_EMAIL_ADDRESS
PosTransactionProperties[1].PosTransactionPropertyValue[1]=user-email@domain.com
PosTransactionProperties[2].PosTransactionPropertyCode[1]=RECEIPT_DELIVERY_METHOD
PosTransactionProperties[2].PosTransactionPropertyValue[1]=EMAIL+PAPER
PosTransactionProperties[3].PosTransactionPropertyCode[1]=CUSTOMER_EMAIL_UPDATED
PosTransactionProperties[3].PosTransactionPropertyValue[1]=true
I would like to create fields and values that look like the following for all events:
RECEIPT_EMAIL_ADDRESS = user-email@domain.com
RECEIPT_DELIVERY_METHOD = EMAIL+PAPER
CUSTOMER_EMAIL_UPDATED = true
I am taking a three (3) step process to achieving my goal: (Still having issues on Step 3)
Step 1 - I did this by extracting the fields using some multivalue field extractions in props.conf and transforms.conf, putting them into two fields (postransactionproperties_PosTransactionPropertyCode and postransactionproperties_PosTransactionPropertyValue)
props.conf
REPORT-trans_PosTransactionProperties = t_PTP_PosTransactionPropertyCode, t_PTP_PosTransactionPropertyValue
transforms.conf
# =-=- PosTransactionProperties[1].PosTransactionPropertyCode[1]=RECEIPT_EMAIL_ADDRESS
[t_PTP_PosTransactionPropertyCode]
REGEX = PosTransactionProperties\[[0-9]+\]\.PosTransactionPropertyCode\[[0-9]+\]\=([^\n]+)\n
FORMAT = postransactionproperties_PosTransactionPropertyCode::$1
MV_ADD = true
REPEAT_MATCH = true
# =-=- PosTransactionProperties[1].PosTransactionPropertyValue[1]=user-email@domain.com
[t_PTP_PosTransactionPropertyValue]
REGEX = PosTransactionProperties\[[0-9]+\]\.PosTransactionPropertyValue\[[0-9]+\]\=([^\n]+)\n
FORMAT = postransactionproperties_PosTransactionPropertyValue::$1
MV_ADD = true
REPEAT_MATCH = true
I then end up with the two mvfields, with all the values lines up in the order they show up in the event.
postransactionproperties_PosTransactionPropertyCode
RECEIPT_EMAIL_ADDRESS
RECEIPT_DELIVERY_METHOD
CUSTOMER_EMAIL_UPDATED
postransactionproperties_PosTransactionPropertyValue
user-email@domain.com
EMAIL+PAPER
true
Step 2 - I then combine the two fields in the order they exist, into a single field matching the values up together, with a separator using an eval statement in the props.conf file
props.conf
EVAL-glue = mvzip(postransactionproperties_PosTransactionPropertyCode,postransactionproperties_PosTransactionPropertyValue,":::")
The result is a field named 'glue' that has values as such:
CUSTOMER_EMAIL_UPDATED:::true
RECEIPT_DELIVERY_METHOD:::EMAIL+PAPER
RECEIPT_EMAIL_ADDRESS:::user-email@domain.com
Step 3 - Which I am having trouble with, I want to now break the values of the field 'glue' up into Key => Value pairs. I am thinking I can do this again with a props.conf and a transforms.conf statement as such:
props.conf
REPORT-trans_glue = t_unglue
transforms.conf
[t_unglue]
SOURCE_KEY = glue
REGEX = ([a-zA-Z0-9\_\-]+):::([a-zA-Z0-9]+)
FORMAT = $1::$2
This unfortunately does not work at all, and I need a little help to either understand why, and how to get it to work, or attack this whole problem in a different fashion all together. (NOTE: I understand the REGEX statement is not 100% correct, but it should have at least matched the 'true' result and it does not).
Thank you
That seems a bit convoluted to me, is there a reason for not using key-value extraction capabilities in transforms.conf directly?
props.conf
[your_sourcetype]
TRANSFORMS-keyvalue = PosTransactionProperties
...
transforms.conf
[PosTransactionProperties]
REGEX = PosTransactionProperties\[\d+\]\.PosTransactionPropertyCode\[\d+\]=(?<_KEY_1>[^\n\r]+)[\n\r]+PosTransactionProperties\[\d+\]\.PosTransactionPropertyValue\[\d+\]=(?<_VAL_1>[^\n\r]+)
This will extract both the field name and the field value using the magic _KEY_n
and _VAL_n
names. Nothing to do in the search itself or in calculated fields, the fields will just be there automatically.
Edit: The reason for your approach not working out is that calculated fields (EVAL-foo
in props.conf) are applied after field extractions, so your transforms.conf rules don't see those fields.