Hi,
I have XML rendered log from sysmon and i need to extract from this log only interesting fields, for example:
Image|UtcTime|ProcessGuid|CommandLine|User|ParentProcessGuid|ParentImage|ParentCommandLine|Hashes
But my conf doesn't work.
What i did wrong and how to fix that?
here is the sample xml
- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
<Provider Name="Microsoft-Windows-Sysmon" Guid="{5770385F-C22A-43E0-BF4C-06F5698FFBD9}" />
<EventID>1</EventID>
<Version>5</Version>
<Level>4</Level>
<Task>1</Task>
<Opcode>0</Opcode>
<Keywords>0x8000000000000000</Keywords>
<TimeCreated SystemTime="2017-03-13T12:16:18.234566900Z" />
<EventRecordID>1098206</EventRecordID>
<Correlation />
<Execution ProcessID="2416" ThreadID="2476" />
<Channel>Microsoft-Windows-Sysmon/Operational</Channel>
<Computer>HOSTNAME</Computer>
<Security UserID="S-1-5-18" />
</System>
- <EventData>
<Data Name="UtcTime">2017-03-13 12:16:18.203</Data>
<Data Name="ProcessGuid">{EF92ED9B-8D92-58C6-0000-0010B2A27B04}</Data>
<Data Name="ProcessId">2832</Data>
<Data Name="Image">C:\Windows\System32\cmd.exe</Data>
<Data Name="CommandLine">"C:\Windows\system32\cmd.exe" /c type "C:\ProgramData\****.txt"</Data>
<Data Name="CurrentDirectory">c:\program files\*****\</Data>
<Data Name="User">NT AUTHORITY\SYSTEM</Data>
<Data Name="LogonGuid">{****************************}</Data>
<Data Name="LogonId">0x3e7</Data>
<Data Name="TerminalSessionId">0</Data>
<Data Name="IntegrityLevel">System</Data>
<Data Name="Hashes">SHA1=0F3C4FF28F354AEDE2,MD5=5746BD7E255DD61,SHA256=DB06C3534964E3FC79D0CA336F4A0FE724B75AAFF386,IMPHASH=D00585440EB0A</Data>
<Data Name="ParentProcessGuid">{**************************}</Data>
<Data Name="ParentProcessId">1564</Data>
<Data Name="ParentImage">C:\Program Files\****.exe</Data>
<Data Name="ParentCommandLine">"C:\Program Files\******" 1452</Data>
</EventData>
+ <RenderingInfo Culture="en-US">
<Message> **************************************************************</Message>
<Level>Information</Level>
<Task>Process Create (rule: ProcessCreate)</Task>
<Opcode>Info</Opcode>
<Channel />
<Provider />
<Keywords />
</RenderingInfo>
</Event>
And this is my conf:
inputs.conf
[WinEventLog://ForwardedEvents]
disabled = false
start_from = oldest
current_only = 0
checkpointInterval = 5
renderXml = true
suppress_text = 1
index = sysmon
sourcetype=XmlWinEventLog:Microsoft-Windows-Sysmon/Operational
whitelist1 = 1,5,6
props.conf
[source::WinEventLog://ForwardedEvents]
TRANSFORMS-setnull = sysmon-setnull
TRANSFORMS-keep = sysmon-keep
transforms.conf
[sysmon-setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[sysmon-keep]
REGEX = (?i)Name=".*(Image|UtcTime|ProcessGuid|CommandLine|User|ParentProcessGuid|ParentImage|ParentCommandLine|Hashes)"
DEST_KEY = queue
FORMAT = indexQueue
The way to do this is to use SEDCMD
to replace the undesired parts with nothing:
https://docs.splunk.com/Documentation/Splunk/latest/Data/Anonymizedata
The configuration that you've is for event filtering, means if an (whole) event is matching a regex, drop the event altogether. The configuration that you're looking for is data masking where you can replace all the not-required lines with blank.
I'm guessing that your xml <Event>
has section <EventData>
and that's the only thing you want to ingest and drop everything else. So give this a try (inputs.conf can stay the same)
props.conf (on indexer/heavy forwarder)
[source::WinEventLog://ForwardedEvents]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\s*\<EventData\>)
SEDCMD-removeheader=s/^(\s*\<Event xmlns(.+[\r\n]*)+)//
SEDCMD-removefooter=s/(\s*\<RenderingInfo (.+[\r\n]*)+)//
Hi, somesoni2
Yes you right, I need only EventData section.
I've try props.conf on heavy forwarder as you describe, but it doesnt work. Headers and footers don't remove.
I've updated the regex for removefooter.
How are you getting the data into heavy forwarder, from universal forwarder? Were heavy forwarder restarted after making the change? Try keep these configurations on Universal forwarder.
I found what was wrong.
In input.conf we set "renderXml = true" . That's why props.conf doesn't apply to source::WinEventLog://ForwardedEvents. Cause source::WinEventLog://ForwardedEvents doesn't exist !
When i change it to "renderXml = false", filter start working!
But, i still need get this events in XML. Is there any way to do that?
"How are you getting the data into heavy forwarder, from universal forwarder?"
No, we collect all events by Windows Event collecor server in ForwardedEvents log. On the same VM we deploy heavy forwarder + Windows_TA addon with all necessary conf.
Hi borshoff,
you cannot filter your events to take only a part of them (only selected fields), You can filter events to take (or discard) all (full) events that match a regex, the only way to limit the dimensions of your events is to put a limit to the number of characters to take for each event (see limits.conf).
Bye.
Giuseppe