Splunk Search

Multivalue field combining and dynamic field creation from extraction of values

jmsiegma
Path Finder

Starting with the data in an event:

Lines in Single Event:

PosTransactionProperties[1].PosTransactionPropertyCode[1]=RECEIPT_EMAIL_ADDRESS
PosTransactionProperties[1].PosTransactionPropertyValue[1]=user-email@domain.com
PosTransactionProperties[2].PosTransactionPropertyCode[1]=RECEIPT_DELIVERY_METHOD
PosTransactionProperties[2].PosTransactionPropertyValue[1]=EMAIL+PAPER
PosTransactionProperties[3].PosTransactionPropertyCode[1]=CUSTOMER_EMAIL_UPDATED
PosTransactionProperties[3].PosTransactionPropertyValue[1]=true

I would like to create fields and values that look like the following for all events:

RECEIPT_EMAIL_ADDRESS = user-email@domain.com
RECEIPT_DELIVERY_METHOD = EMAIL+PAPER
CUSTOMER_EMAIL_UPDATED = true

I am taking a three (3) step process to achieving my goal: (Still having issues on Step 3)

  1. Extract the initial values from the original lines as multi-valued values
  2. Combine the values in a single field
  3. Break that field into parts, that will create the new field=values format I am desiring

Step 1 - I did this by extracting the fields using some multivalue field extractions in props.conf and transforms.conf, putting them into two fields (postransactionproperties_PosTransactionPropertyCode and postransactionproperties_PosTransactionPropertyValue)

props.conf
REPORT-trans_PosTransactionProperties = t_PTP_PosTransactionPropertyCode, t_PTP_PosTransactionPropertyValue

transforms.conf
# =-=- PosTransactionProperties[1].PosTransactionPropertyCode[1]=RECEIPT_EMAIL_ADDRESS
[t_PTP_PosTransactionPropertyCode]
REGEX = PosTransactionProperties\[[0-9]+\]\.PosTransactionPropertyCode\[[0-9]+\]\=([^\n]+)\n
FORMAT = postransactionproperties_PosTransactionPropertyCode::$1 
MV_ADD = true
REPEAT_MATCH = true

# =-=- PosTransactionProperties[1].PosTransactionPropertyValue[1]=user-email@domain.com
[t_PTP_PosTransactionPropertyValue]
REGEX = PosTransactionProperties\[[0-9]+\]\.PosTransactionPropertyValue\[[0-9]+\]\=([^\n]+)\n
FORMAT = postransactionproperties_PosTransactionPropertyValue::$1 
MV_ADD = true
REPEAT_MATCH = true

I then end up with the two mvfields, with all the values lines up in the order they show up in the event.

postransactionproperties_PosTransactionPropertyCode 
RECEIPT_EMAIL_ADDRESS   
RECEIPT_DELIVERY_METHOD 
CUSTOMER_EMAIL_UPDATED

postransactionproperties_PosTransactionPropertyValue
user-email@domain.com   
EMAIL+PAPER 
true

Step 2 - I then combine the two fields in the order they exist, into a single field matching the values up together, with a separator using an eval statement in the props.conf file

props.conf
EVAL-glue = mvzip(postransactionproperties_PosTransactionPropertyCode,postransactionproperties_PosTransactionPropertyValue,":::")

The result is a field named 'glue' that has values as such:

CUSTOMER_EMAIL_UPDATED:::true
RECEIPT_DELIVERY_METHOD:::EMAIL+PAPER
RECEIPT_EMAIL_ADDRESS:::user-email@domain.com

Step 3 - Which I am having trouble with, I want to now break the values of the field 'glue' up into Key => Value pairs. I am thinking I can do this again with a props.conf and a transforms.conf statement as such:

props.conf
REPORT-trans_glue = t_unglue

transforms.conf
[t_unglue]
SOURCE_KEY = glue
REGEX = ([a-zA-Z0-9\_\-]+):::([a-zA-Z0-9]+)
FORMAT = $1::$2

This unfortunately does not work at all, and I need a little help to either understand why, and how to get it to work, or attack this whole problem in a different fashion all together. (NOTE: I understand the REGEX statement is not 100% correct, but it should have at least matched the 'true' result and it does not).

Thank you

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

That seems a bit convoluted to me, is there a reason for not using key-value extraction capabilities in transforms.conf directly?

props.conf

[your_sourcetype]
TRANSFORMS-keyvalue = PosTransactionProperties
...

transforms.conf

[PosTransactionProperties]
REGEX = PosTransactionProperties\[\d+\]\.PosTransactionPropertyCode\[\d+\]=(?<_KEY_1>[^\n\r]+)[\n\r]+PosTransactionProperties\[\d+\]\.PosTransactionPropertyValue\[\d+\]=(?<_VAL_1>[^\n\r]+)

This will extract both the field name and the field value using the magic _KEY_n and _VAL_n names. Nothing to do in the search itself or in calculated fields, the fields will just be there automatically.

Edit: The reason for your approach not working out is that calculated fields (EVAL-foo in props.conf) are applied after field extractions, so your transforms.conf rules don't see those fields.

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...