This is an example of my source:
/frameworks/app_console-ui_v656_web_0/runs/latest/errors.stdout
I am using the following props and transforms for my test case:
## props.conf
[mytest]
TRANSFORMS-extract-source = extract-source
DATETIME_CONFIG = CURRENT
NO_BINARY_CHECK = true
category = Custom
pulldown_type = true
SHOULD_LINEMERGE = false
disabled = false
## transforms.conf
[extract-source]
SOURCE_KEY = MetaData:Source
REGEX = ^\/frameworks\/app_console-ui_v([\d]+)_([^_]+)_
FORMAT = job_id::$2.$1
WRITE_META = true
I have confirmed that the regex used is correct, but it's not working if "SOURCE_KEY = MetaData:Source" is used.
Looks like if an anchor is used at the beginning of the regex, it needs to include source:: as this is how it's written in the bucket's Sources.data metadata file.
This revised stanza appears to be working for me now:
## transforms.conf
[extract-source]
SOURCE_KEY = MetaData:Source
REGEX = ^source::\/frameworks\/app_console-ui_v([\d]+)_([^_]+)_
FORMAT = job_id::$2.$1
WRITE_META = true
Hope this information will help someone encountering similar issues.
Thank you, I was going nuts trying to figure out where source:: was coming from.
Looks like if an anchor is used at the beginning of the regex, it needs to include source:: as this is how it's written in the bucket's Sources.data metadata file.
This revised stanza appears to be working for me now:
## transforms.conf
[extract-source]
SOURCE_KEY = MetaData:Source
REGEX = ^source::\/frameworks\/app_console-ui_v([\d]+)_([^_]+)_
FORMAT = job_id::$2.$1
WRITE_META = true
Hope this information will help someone encountering similar issues.