Multivalue field combining and dynamic field creation from extraction of values

[jmsiegma]

Starting with the data in an event:

Lines in Single Event:

PosTransactionProperties[1].PosTransactionPropertyCode[1]=RECEIPT_EMAIL_ADDRESS
PosTransactionProperties[1].PosTransactionPropertyValue[1]=user-email@domain.com
PosTransactionProperties[2].PosTransactionPropertyCode[1]=RECEIPT_DELIVERY_METHOD
PosTransactionProperties[2].PosTransactionPropertyValue[1]=EMAIL+PAPER
PosTransactionProperties[3].PosTransactionPropertyCode[1]=CUSTOMER_EMAIL_UPDATED
PosTransactionProperties[3].PosTransactionPropertyValue[1]=true

I would like to create fields and values that look like the following for all events:

RECEIPT_EMAIL_ADDRESS = user-email@domain.com
RECEIPT_DELIVERY_METHOD = EMAIL+PAPER
CUSTOMER_EMAIL_UPDATED = true

I am taking a three (3) step process to achieving my goal: (Still having issues on Step 3)

1. Extract the initial values from the original lines as multi-valued values
2. Combine the values in a single field
3. Break that field into parts, that will create the new field=values format I am desiring

Step 1 - I did this by extracting the fields using some multivalue field extractions in props.conf and transforms.conf, putting them into two fields (postransactionproperties_PosTransactionPropertyCode and postransactionproperties_PosTransactionPropertyValue)

props.conf
REPORT-trans_PosTransactionProperties = t_PTP_PosTransactionPropertyCode, t_PTP_PosTransactionPropertyValue

transforms.conf
# =-=- PosTransactionProperties[1].PosTransactionPropertyCode[1]=RECEIPT_EMAIL_ADDRESS
[t_PTP_PosTransactionPropertyCode]
REGEX = PosTransactionProperties\[[0-9]+\]\.PosTransactionPropertyCode\[[0-9]+\]\=([^\n]+)\n
FORMAT = postransactionproperties_PosTransactionPropertyCode::$1 
MV_ADD = true
REPEAT_MATCH = true

# =-=- PosTransactionProperties[1].PosTransactionPropertyValue[1]=user-email@domain.com
[t_PTP_PosTransactionPropertyValue]
REGEX = PosTransactionProperties\[[0-9]+\]\.PosTransactionPropertyValue\[[0-9]+\]\=([^\n]+)\n
FORMAT = postransactionproperties_PosTransactionPropertyValue::$1 
MV_ADD = true
REPEAT_MATCH = true

I then end up with the two mvfields, with all the values lines up in the order they show up in the event.

postransactionproperties_PosTransactionPropertyCode 
RECEIPT_EMAIL_ADDRESS   
RECEIPT_DELIVERY_METHOD 
CUSTOMER_EMAIL_UPDATED

postransactionproperties_PosTransactionPropertyValue
user-email@domain.com   
EMAIL+PAPER 
true

Step 2 - I then combine the two fields in the order they exist, into a single field matching the values up together, with a separator using an eval statement in the props.conf file

props.conf
EVAL-glue = mvzip(postransactionproperties_PosTransactionPropertyCode,postransactionproperties_PosTransactionPropertyValue,":::")

The result is a field named 'glue' that has values as such:

CUSTOMER_EMAIL_UPDATED:::true
RECEIPT_DELIVERY_METHOD:::EMAIL+PAPER
RECEIPT_EMAIL_ADDRESS:::user-email@domain.com

Step 3 - Which I am having trouble with, I want to now break the values of the field 'glue' up into Key => Value pairs. I am thinking I can do this again with a props.conf and a transforms.conf statement as such:

props.conf
REPORT-trans_glue = t_unglue

transforms.conf
[t_unglue]
SOURCE_KEY = glue
REGEX = ([a-zA-Z0-9\_\-]+):::([a-zA-Z0-9]+)
FORMAT = $1::$2

This unfortunately does not work at all, and I need a little help to either understand why, and how to get it to work, or attack this whole problem in a different fashion all together. (NOTE: I understand the REGEX statement is not 100% correct, but it should have at least matched the 'true' result and it does not).

Thank you

[martin_mueller]

That seems a bit convoluted to me, is there a reason for not using key-value extraction capabilities in transforms.conf directly?

props.conf

[your_sourcetype]
TRANSFORMS-keyvalue = PosTransactionProperties
...

transforms.conf

[PosTransactionProperties]
REGEX = PosTransactionProperties\[\d+\]\.PosTransactionPropertyCode\[\d+\]=(?<_KEY_1>[^\n\r]+)[\n\r]+PosTransactionProperties\[\d+\]\.PosTransactionPropertyValue\[\d+\]=(?<_VAL_1>[^\n\r]+)

This will extract both the field name and the field value using the magic _KEY_n and _VAL_n names. Nothing to do in the search itself or in calculated fields, the fields will just be there automatically.

Edit: The reason for your approach not working out is that calculated fields (EVAL-foo in props.conf) are applied after field extractions, so your transforms.conf rules don't see those fields.