Field extraction fails to extract all fields

[bryancock]

Hi All

I'm hoping someone might have an insight into a problem I'm having extracting fields from some XML based event data -- essentially I can only extract the first of 2 potential fields; the second field fails to be extracted and made available in the search app.

Example of the data I'm indexing:


<metrics>
<metric>
<name>diskavailablespaceC</name>
<value>1</value>
</metric>
<metric>
<name>diskavailablespaceE</name>
<value>7</value>
</metric>
</metrics>


My config files:

props.conf

[Power_test1]
TRANSFORMS-disk_available = disk_available


transforms.conf

[disk_available]
REGEX = (?i)<name>diskavailablespace(\w)</name>\s*?<value>(\d)</value>
FORMAT = $1::$2
WRITE_META = true


Some observations:

• My expectation is that 2 fields should be extracted: C = 1 and E = 7.
• Using the above, only a single field name 'C' is extracted from my indexed event data with value 1.
• If I swap the order of the <metric> tags around, I extract a single field 'E' with value 7, suggesting to me that the regex is either failing or not being applied beyond the first match (??).
• My regular expression, when tested in an external tool, seems to check out OK as it produces 2 matches of (2) groups --> one match for content of each <metric> tag.
• I've added REPEAT_MATCH = true attribute with no change in behaviour.

I would appreciate any suggestions as to what may be going on, or what I might do to achieve the expected field extraction.

Many Thanks, B

[Ayn]

Two things:

1. You really shouldn't be using index-time extractions (TRANSFORMS) unless you really know what you're doing and have a good reason for it. Search-time extractions give better performance and flexibility.
2. Only one value will be extracted because this is default behaviour when encountering more than one value for a field in an event. This behaviour can be toggled using the MV_ADD directive in transforms.conf. From the docs:

``

MV_ADD = [true|false]
* NOTE: This attribute is only valid for search-time field extractions.
* Optional. Controls what the extractor does when it finds a field which already exists.
* If set to true, the extractor makes the field a multivalued field and appends the 
* newly found value, otherwise the newly found value is discarded.
* Defaults to false