Hi,
We are capturing a custom log from Windows event viewer using Splunk forwarder. Most of the fields are extracted properly in field:value
format, except one where the data itself contains sub-fields with their own values.
I tried creating reg-ex field extractions but it only identified in 1 or 2 events and keeps failing for other.
e.g. Below is a sanitized version of the event we see. All fields from LogName
to Keywords
are parsed correctly, except when it reached Message
. This field itself contains other items e.g. Client
, Computer
, or Action
.
What is the best way to extract these fields as well?
08/12/2016 10:23:03 AM
LogName=xxxxxxxxx
SourceName=xxxxxxxxx
EventCode=3
EventType=4
Type=Information
ComputerName=Name.x.y.z.local
User=aaaa
Sid=S-1-5-7
SidType=5
TaskCategory=%1
OpCode=None
RecordNumber=1234567
Keywords=Classic
Message=AD object was successfully modified.
Client Computer : w.x.y.z
Object DN : CN=Username,OU=Junior,DC=x,DC=y,DC=z,DC=local
Object Class : user
Object GUID : CN=Username,OU=Junior,DC=x,DC=y,DC=z,DC=local Attribute Name : logonCount
Action : Update
Old Value : 1234
New Value : 1235
Request ID : {84E5C0C7-84E5C0C7-84E5C0C7-84E5C0C7}
You can install the Windows_TA
app and it should do field extractions for you.
Hi woodcock,
We have Splunk_TA_windows installed on both the Search head as well as the forwarder sending these events, but the Message field still the same. Entire Message is clubbed into one big string.
We also tried moving the monitor stanza on the universal forwarder from the system/local/inputs.conf to inputs.conf for windows TA, but it still behaves the same way.
~ Abhi
I'm having the same issue with the windows TA installed as well.
Let's make sure that we are talking about the same things; see here:
https://answers.splunk.com/answers/504719/splunk-for-windows-app-on-standalone-instance.html
If you looking to extract this in the search screen, you could use the extract
command.
... | extract kvdelim=" : " pairdelim="\n"
http://docs.splunk.com/Documentation/Splunk/6.4.2/SearchReference/Extract
If you want to extract only the string after message= but not the other lines, you have to insert \n at the end of your regex, see below:
Message=(?.*)\n
if you want to try this regex use https://regex101.com/
Bye.
Giuseppe
it's the same thing :
Client Computer : (? .)\n
Object DN : (? .)\n
Object Class : (? .)\n
Object GUID : (? .)\n
Action : (? .)\n
Old Value : (? .)\n
New Value : (? .)\n
Request ID : (? .)
You have to extract every field inserting \n at the end
Bye.
Giuseppe
You can do this
index=foo | rex field=ORIGINAL_FILED_NAME "(?<NEW_FILED_NAME>\w+)"
This will do a regular expression and extract data from your first field (ORIGINAL_FIELD_NAME) and create a second field (NEW_FILED_NAME) with the extracted value
I'm not sure what second field you want extracted, give us more details and I'll write the regex for you
Hi skoelpin,
Thank you for the response.
From the "Message" field in above example, we would like to extract all the sub-field, i.e. Client Computer, Object DN, Object Class, Object GUID, Action, Old Value, New Value and Request ID. It will become much easier to create dashboards and alerts if all these are their own field:value pair instead of being part of original "Message" field.
I would recommend extracting these sub-fields at search time in your props.conf
So you would go to your props.conf
and go to your stanza and add this
[YOUR_SOURCETYPE]
EXTRACT-GUID = ((?<=CN\=|OU\=|DC\=)\w+)
Hi skoelpin,
I tried this but didn't work. Just to make sure, this needs to be added under the system/local/props.conf correct? because we also have Splunk_TA_windows addon installed which has it's own props.conf.
I also came across another answer but again, none of the changes take effect.
https://answers.splunk.com/answers/49310/field-extraction-in-message-field-of-windows-event-log.html
I am not sure If I am editing/adding extractions at the wrong location? or if it's because the source/sourcetype contains white spaces, although I tried adding a wildcard * but in vain.
source=[WinEventLog:XXX Y Z]
Based on the fields currently being extracted for windows event's, it looks like the props from Windows TA is contributing as well. I am not sure if this takes precedence over /local/props.conf or if both work together.
Is there a way to figure out which extractions/transforms and from which location are being used for a specific sourcetype.