Loglines with standard fields followed by kv pairs.

[davidatpinger]

I've got a bunch of loglines that are very boring key value pairs - comma separated list of key=value. (So far, so good.) The first few (let's say three for example purposes) are always the same, and the rest (the number is dynamic) are variable depending on what's in the first three.

For the purpose of reducing the amount of data I'm ingesting, I'd like to just list the values for the first three columns and not identify the keys. That is, I'd take a line that starts out like this:

a=1,b=2,c=3,d=4,e=5,f=6

and simplify it to

1,2,3,d=4,e=5,f=6

(That only saves a few chars in this example, but the full dataset has more verbose field names, so it adds up to a lot!)

I thought I could just write a block in transforms.conf like this:

[foo]
REGEX = ^(?<a>[^,]*),(?<b>[^,]*),(?<c>[^,]*),

and then apply it in props.conf. I've done that but I'm now getting no fields extracted at all. (I should mention that this is an index time extraction, so I'm doing it with TRANSFORMS-foo.)

I'm at a bit of a loss about how to figure out why this isn't working. I kinda expected that this might prevent extraction of the kv pairs past the REGEX, but it seems to now do nothing. Suggestions about how to sort this out?

[maciep · Accepted solution]

First question is why you're doing that at index time? Seems easy enough to extract at search time.

That said, I wonder if maybe you need to add WRITE_META = true to your stanza in transforms.conf? It defaults to false. And it seems to be required for index time extractions.

WRITE_META = [true|false]
* NOTE: This attribute is only valid for index-time field extractions.
* Automatically writes REGEX to metadata.
* Required for all index-time field extractions except for those where DEST_KEY = _meta (see    the description of the DEST_KEY attribute, below)
* Use instead of DEST_KEY = _meta.
* Defaults to false.

[davidatpinger]

One correction - it does seem to be extracting the kv pairs after the 'fixed' section.