Extract fields from Windows Event Log Message

[dw385]

I have events that do not extract the fields from the message field by default. I'm trying to setup props/transforms to auto extract the fields on search. I believe I have the props and transforms setup properly however it's not working. I think the issue is the regex I'm using and possibly the first 2 lines of the message data.

Props:

[source::WinEventLog:Directory Service]
REPORT-dsfields = ds_fields

Transforms:

[ds_fields]
SOURCE_KEY = Message
REGEX = ([^:]+):\n(.*[\S])
FORMAT = $1::$2
MV_ADD = true
KEEP_EMPTY_VALS = true

The regex appears to extract the fields properly with the exception of the first line which I don't need. I also tried this regex

(?:.*\n){2}([^:]+):\n(.*[\S])

Which also appears to work to exclude the first two lines. However the combination doesn't extract fields when searching. If I extract a single value in the same props and transforms it works so I believe the files are configured properly.

I don't care if the first line (Internal Event:) is extracted or not. The goal is to extract the field value pairs in the rest of the message.

The message field data looks like:

Internal event: A client issued a search operation with the following options. 

Client:
10.215.215.215:61759 
Starting node:
DC=domain,DC=Domain,DC=Domain,DC=local 
Filter:
 (accountNameHistory=*[DG]*)  
Search scope:
subtree 
Attribute selection:
accountNameHistory,employeeType,title,info,groupType,distinguishedName,objectGUID,sAMAccountType,departmentNumber,memberOf,objectSid,objectClass
Server controls:

Visited entries:
1556 
Returned entries:
8 
Used indexes:
Ancestors_index:3356:N; 
Pages referenced:
16745 
Pages read from disk:
31 
Pages preread from disk:
9 
Clean pages modified:
0 
Dirty pages modified:
0 
Search time (ms):
313 
Attributes Preventing Optimization:
accountNameHistory  
User:
Domain\some-user-account

[adonio]

hello there,
the configurations you need are pre-built and packaged nicely on the Splunk_TA_microsoft_ad
download here: https://splunkbase.splunk.com/app/3207/
install it on the forwarder of your domain controller and you can remove the props and transforms
read more here:
http://docs.splunk.com/Documentation/MSApp/1.4.1/MSInfra/WhataSplunkAppforWindowsInfrastructuredeplo...
and here:
http://docs.splunk.com/Documentation/MSApp/1.4.1/MSInfra/ConfigureActiveDirectoryauditpolicy
hope it helps

[dw385]

I have that deployed to my DCs and on the SHs. The data is collected but some of the logs in Directory Services is in a header over value format. It seems none of values are extracted properly (not just for this particular event I need). I don't think I see anything in the AD TA that seems to suggest there is anything for extracting the fields from the message.

[adonio]

do you have the TA on the indexer as well?

[dw385]

Yes, it's on the indexers as well. Latest version (1.0) with no customization to it. This is the same for the Windows TA as well, although not the latest version, I am a release or two behind on that one.

[adonio]

can you verify the source of the data you are trying to transform?
i see in the exchange app i have access to that the source is "Directory_Service" and not
"Directory Service"
in any case, its odd, i see all extractions working fine on the environment i have access to
hope it helps

[Richfez]

I changed the text of the event code to "code" so the asterisks won't be converted to italics. It did not change the carriage returns, but I don't know if this is the way it is supposed to be. Can you confirm it still looks like the event as you see it in the original place?

Thanks!

[dw385]

Thank you, the example looks correct.