How do I configure Splunk to recognize my custom delimiter for proper field extraction?

[emamedov]

I currently have a log statement which has a custom delimiter: {|}

Where an example log statement would look like:

Oct-13 12:17:13 | INFO| [Logger:152] Message{|}Activity1{|}userDeletedProfile{|}John Smith{|}Smith Securities{|}Test1{|}5512{|}324166{|}552341{|}260

However, when I try to conduct a field extraction where DELIMS = "{|}", the fields aren't being extracted properly. However, testing the above log statement in another application that is capable of delimiting yields successful results.

[gokadroid]

Hi @emamedov

I tried to extract fields using the Field Extractor and it did extract the fields with {|} however for some odd reason it considered pipe in between as a field too. So all of below in " " are each field as they appear on my field extractor.

"Oct-13 12:18:13 | INFO| [Logger:152] Message"  "|"  "Activity1" "|" "userDeletedProfile" "|" "John" "|" "KMI SECUR" "|" "Test1" "|" "5512" "|" "324166" "|" "552341" "|" "260"

If you are not fine with above then alternatively you can use below regex in "write my own regular expression" in FieldExtractor or during search time:

^(?< dateTime>[^|]+)|\s+?(?< loggerLevel>[^|]+)|\s+?(?< message>[^{]+){|}(?< activity>[^{]+){|}(?< profile>[^{]+){|}(?< userName>[^{]+){|}(?< securities>[^{]+){|}(?< test>[^{]+){|}(?< id1>[^{]+){|}(?< id2>[^{]+){|}(?< id3>[^{]+){|}(?< id4>[^{]+)

Note Please remove space in each of the tags above < dateTime> < loggerLevel> and so on

[gcusello]

You have to extract fields using multiple regex.
I cannot use my pc now, Tomorrow morning I'll send you an example.
Bye.
Giuseppe

[gcusello]

in props. conf
REPORT-myfields = myfields_Fields

in transforms.conf
[myfields_Fields]
REGEX = ](.){|}(.){|}(.){|}(.){|}(.){|}(.){|}(.){|}(.){|}(.){|}(.)
FORMAT = Field1::"$1" Field2::"$2" Field3::"$3" Field4::"$4" Field5::"$5" Field6::"$6" Field7::"$7" Field8::"$8" Field9::"$9" Field10::"$10"
WRITE_META = true

Bye.
Giuseppe

[rjthibod]

I think you cannot use the DELIMS setting to use ordered groups of characters as delimeters. Reading http://docs.splunk.com/Documentation/Splunk/latest/Admin/Transformsconf seems to say that you can only specify a single character delimeter.

Instead of DELIM you will probably have to specify the fields/delimeters with REGEX-based extractions.

[rjthibod]

The community probably needs more info to help. Can you share more details about the settings in props.conf that are associated with this sourcetype?