Field Extraction in Message field of Windows Event Log

[dturner83]

Hello,

I have an application sending logs to the windows event log with a lognamename of ErrorLogs. The error log looks like this:

05/30/2012 03:17:42 PM

LogName=ErrorLogs

SourceName=LogDataManager

EventCode=0

EventType=4

Type=Information

ComputerName=DEVSERVICE

TaskCategory=%1

OpCode=Info

RecordNumber=31

Keywords=Classic

Message=ApplicationName: EventLogTestWriter

Severity: Informational

Error: Added newline before ApplicationName

StackTrace: N/A

All fields extract well in the autoextraction in search except for the first field 'ApplicationName:' Any ideas? I tried a new line break and while it does send ApplicationName to a new line I still don't get the field to extract.

[kristian_kolb]

All WinEvent logs go through the parsing rules defined in $SPLUNK_HOME/etc/system/local/props.conf and $SPLUNK_HOME/etc/system/local/transforms.conf.

These state the following;

props.conf:

[source::WinEventLog...]
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD=30
LINE_BREAKER = ([\r\n](?=\d{2}/\d{2}/\d{2,4} \d{2}:\d{2}:\d{2} [aApPmM]{2}))
REPORT-MESSAGE = wel-message, wel-eq-kv, wel-col-kv
KV_MODE=none

etc, etc

transforms.conf:

[wel-message]
REGEX = (?sm)^(?<_pre_msg>.+)\nMessage=(?<Message>.+)$
CLEAN_KEYS = false

[wel-eq-kv]
SOURCE_KEY = _pre_msg
DELIMS     = "\n","="
MV_ADD     = true

[wel-col-kv]
SOURCE_KEY = Message
REGEX      = \n([^:\n\r]+):[ \t]++([^\n]*)
FORMAT     = $1::$2
MV_ADD     = true

This means that anything with the source of WinEventLog:somthing will be run through the three field extracting transforms wel-message, wel-eq-kv and wel-col-kv. In that order.

wel-message splits the event into two fields, _pre_msg and Message.
wel-eq-kv splits the _pre_msg into field/value pairs based on 'equals' (=).
wel-col-kv splits the Message into field/value pairs based on colons (:).

However the REGEX in wel-col-kv requires that a newline preceeds the first capture group, and that newline does not exist in your first row.

Perhaps you could/should create new props.conf/transforms.conf stanzas in the /$SPLUNK_HOME/etc/system/local directory, which would be very similar, but the differently named. The props-stanza (from which the transforms are called) should be more specific, i.e. [WinEventLog:ErrorLogs], and the transforms that are called through the REPORT should be called welel-message, welel-eq-kv and welel-col-kv respectively (. Then the REGEX in welel-col-kv should have the leading newline made optional (by a question mark);

 [welel-col-kv]
 SOURCE_KEY = Message
 REGEX      = \n?([^:\n\r]+):[ \t]++([^\n]*)
 FORMAT     = $1::$2
 MV_ADD     = true

Hope this helps, at least a little bit.

Kristian

[kristian_kolb]

So it's pretty much like all the other WinEventLogs, i.e. Message contains everything from the Message= to the end of the event?

The answer is going to be to long for a comment. See answer below.

/k

[dturner83]

Message itself contains all of ApplicationName through StackTrace inside that field in the eventlog. A splunk search on this yields that the Severity, Error, and StackTrace Fields show up in the 22 interesting fields, I assume being autoextracted, but ApplicationName itself does not get extracted.

[kristian_kolb]

How does it not 'extract well'?
Is there a field called ApplicationName?
Does it contain anything?
Too much? Too little?

What is the value of the Message field?

/k