In order to reduce the amount of data being indexed, I am using a sed script to strip away all XML tags and to format the data as key=value
. Sometimes there are multiple key-value pairs like in the below sample event.
2013-11-07 03:20:24,637 Outgoing UserId="555555555555" Type="Main" To="user@example.com" Key="1" Text="Value 1" Key="2" Text="Value 2" Key="3" Text="Value 3" SessionId="1000"
I want the search to include all occurrences of multiple key-value pairs (i.e. search..| stats count by Key
). Currently, I will only get Key=1 in my results. I have added MV_ADD
to transforms.conf but it is not working. Below is my current configuration.
props.conf
[source::/path/to/test.txt]
SHOULD_LINEMERGE = False
SEDCMD-trim = s/<([^\s\>]*)[^\>]*\>([^<]*)\<\/\1\>/ \1="\2"/g
REPORT-mvfield = mv-field
transforms.conf
[mv-field]
MV_ADD = true
What am I missing?
Update:
Based on this post I was able to determine that I needed to add a transforms for each field.
[mv-field-key]
REGEX = Key=\"(.+?)\"
FORMAT = Key::$1
MV_ADD = true
[mv-field-text]
REGEX = \bText\b=\"(.+?)\"
FORMAT = Text::$1
MV_ADD = true
Is it possible to have a single stanza so that I don't have to specify each field that I want to apply MV_ADD?
Yes, you can do this instead of individual extractions:
[mv-field]
REGEX = (?<_KEY_1>\w+)=\"(?<_VAL_1>.*?)\"
MV_ADD = true
_KEY_1
and _VAL_1
have special meanings in transforms.conf
Edit: Change regexes in the named capture groups to fit your needs.
Yes, you can do this instead of individual extractions:
[mv-field]
REGEX = (?<_KEY_1>\w+)=\"(?<_VAL_1>.*?)\"
MV_ADD = true
_KEY_1
and _VAL_1
have special meanings in transforms.conf
Edit: Change regexes in the named capture groups to fit your needs.
Perfect, thanks!