Automatic search-time field extraction: Need equals sign??

[mfeeny1]

I recently received a request/complaint from one of our users that a certain field ("Trace ID") was being extracted from logs on one set of hosts, but not from those on another set.

I have concluded that the reason for this discrepancy is the difference in formatting of this “field” in the respective logfiles, as follows...

On "good" hosts: traceId=a804f6ea-8c8c-4d2b-8573-93c9629f521a

On bad" hosts: [Trace Id:000fd761-83b2-46b8-b200-ce4510350cae]

Is my conclusion correct?? Is an equals sign necessary for Splunk to automatically extract a given field at search time?? (By "automatically", I mean, as described in the doc's, "[Splunk] automatically identifies and extracts the first 50 fields that it finds in the event data that match obvious name/value pairs".

And... I assume that Splunk would not tolerate any spaces in the automatically extracted Field Name or Field Value, right? It would consider only non-spaced text strings on either side of the equals sign, yeah?

Thx for any confirmation/clarification!

mfeeny1

[mfeeny1]

Thx for the quick, thorough response - appreciated!

I will look into adding a DELIMS block, UNLESS... I can convince the logfile owners to reformat their field to use "traceId=1a2b3c...".

(It can't hurt to ask, right???)

Regards,

mfeeny1

[bbingham · Accepted solution]

For default settings without any configuration, yes you need an equals sign. You can always use any regex string to extract the key value pairs (example: $1::$2). You can also add any character to a list for key-value extraction. In transforms.conf, you'll just simply need to add a DELIMS block to your stanza:

Here's a block from the docs: Transforms.conf

DELIMS = <quoted string list>
* NOTE: This attribute is only valid for search-time field extractions.
* IMPORTANT: If a value may contain an embedded unescaped double quote character, 
  such as "foo"bar", use REGEX, not DELIMS. An escaped double quote (\") is ok.
* Optional. Used in place of REGEX when dealing with delimiter-based field extractions, 
  where field values (or field/value pairs) are separated by delimiters such as colons, 
  spaces, line breaks, and so on.
* Sets delimiter characters, first to separate data into field/value pairs, and then to 
  separate field from value.
* Each individual character in the delimiter string is used as a delimiter to split the event.
* Delimiters must be quoted with " " (use \ to escape).
* When the event contains full delimiter-separated field/value pairs, you enter two sets of 
  quoted characters for DELIMS: 
    * The first set of quoted delimiters extracts the field/value pairs.
    * The second set of quoted delimiters separates the field name from its corresponding
      value.
* When the event only contains delimiter-separated values (no field names) you use just one set
  of quoted delimiters to separate the field values. Then you use the FIELDS attribute to
  apply field names to the extracted values (see FIELDS, below).
    * Alternately, Splunk reads even tokens as field names and odd tokens as field values.
* Splunk consumes consecutive delimiter characters unless you specify a list of field names.
* The following example of DELIMS usage applies to an event where field/value pairs are 
  seperated by '|' symbols and the field names are separated from their corresponding values 
  by '=' symbols:
    [pipe_eq]
    DELIMS = "|", "="
* Defaults to "".   

check out this file in your splunk install for some samples:

[$SPLUNK_HOME]/etc/system/README/transforms.conf.example