Splunk Search

How to filter XmlWinEventLog in Heavy Forwarder with regex?

borshoff
Explorer

Hi,
I have XML rendered log from sysmon and i need to extract from this log only interesting fields, for example:

Image|UtcTime|ProcessGuid|CommandLine|User|ParentProcessGuid|ParentImage|ParentCommandLine|Hashes

But my conf doesn't work.
What i did wrong and how to fix that?

here is the sample xml

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
  <Provider Name="Microsoft-Windows-Sysmon" Guid="{5770385F-C22A-43E0-BF4C-06F5698FFBD9}" /> 
  <EventID>1</EventID> 
  <Version>5</Version> 
  <Level>4</Level> 
  <Task>1</Task> 
  <Opcode>0</Opcode> 
  <Keywords>0x8000000000000000</Keywords> 
  <TimeCreated SystemTime="2017-03-13T12:16:18.234566900Z" /> 
  <EventRecordID>1098206</EventRecordID> 
  <Correlation /> 
  <Execution ProcessID="2416" ThreadID="2476" /> 
  <Channel>Microsoft-Windows-Sysmon/Operational</Channel> 
  <Computer>HOSTNAME</Computer> 
  <Security UserID="S-1-5-18" /> 
  </System>
- <EventData>
  <Data Name="UtcTime">2017-03-13 12:16:18.203</Data> 
  <Data Name="ProcessGuid">{EF92ED9B-8D92-58C6-0000-0010B2A27B04}</Data> 
  <Data Name="ProcessId">2832</Data> 
  <Data Name="Image">C:\Windows\System32\cmd.exe</Data> 
  <Data Name="CommandLine">"C:\Windows\system32\cmd.exe" /c type "C:\ProgramData\****.txt"</Data> 
  <Data Name="CurrentDirectory">c:\program files\*****\</Data> 
  <Data Name="User">NT AUTHORITY\SYSTEM</Data> 
  <Data Name="LogonGuid">{****************************}</Data> 
  <Data Name="LogonId">0x3e7</Data> 
  <Data Name="TerminalSessionId">0</Data> 
  <Data Name="IntegrityLevel">System</Data> 
  <Data Name="Hashes">SHA1=0F3C4FF28F354AEDE2,MD5=5746BD7E255DD61,SHA256=DB06C3534964E3FC79D0CA336F4A0FE724B75AAFF386,IMPHASH=D00585440EB0A</Data> 
  <Data Name="ParentProcessGuid">{**************************}</Data> 
  <Data Name="ParentProcessId">1564</Data> 
  <Data Name="ParentImage">C:\Program Files\****.exe</Data> 
  <Data Name="ParentCommandLine">"C:\Program Files\******" 1452</Data> 
  </EventData>
+ <RenderingInfo Culture="en-US">
  <Message> **************************************************************</Message> 
  <Level>Information</Level> 
  <Task>Process Create (rule: ProcessCreate)</Task> 
  <Opcode>Info</Opcode> 
  <Channel /> 
  <Provider /> 
  <Keywords /> 
  </RenderingInfo>
  </Event>

And this is my conf:

inputs.conf

[WinEventLog://ForwardedEvents]
    disabled = false
    start_from = oldest
    current_only = 0
    checkpointInterval = 5
    renderXml = true
    suppress_text = 1
    index = sysmon
    sourcetype=XmlWinEventLog:Microsoft-Windows-Sysmon/Operational
    whitelist1 = 1,5,6

props.conf

 [source::WinEventLog://ForwardedEvents]
    TRANSFORMS-setnull = sysmon-setnull
    TRANSFORMS-keep = sysmon-keep

transforms.conf

[sysmon-setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[sysmon-keep]
REGEX = (?i)Name=".*(Image|UtcTime|ProcessGuid|CommandLine|User|ParentProcessGuid|ParentImage|ParentCommandLine|Hashes)"
DEST_KEY = queue
FORMAT = indexQueue
0 Karma

woodcock
Esteemed Legend

The way to do this is to use SEDCMD to replace the undesired parts with nothing:

https://docs.splunk.com/Documentation/Splunk/latest/Data/Anonymizedata

0 Karma

somesoni2
SplunkTrust
SplunkTrust

The configuration that you've is for event filtering, means if an (whole) event is matching a regex, drop the event altogether. The configuration that you're looking for is data masking where you can replace all the not-required lines with blank.

I'm guessing that your xml <Event> has section <EventData> and that's the only thing you want to ingest and drop everything else. So give this a try (inputs.conf can stay the same)

props.conf (on indexer/heavy forwarder)

[source::WinEventLog://ForwardedEvents]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\s*\<EventData\>)
SEDCMD-removeheader=s/^(\s*\<Event xmlns(.+[\r\n]*)+)//
SEDCMD-removefooter=s/(\s*\<RenderingInfo (.+[\r\n]*)+)//

borshoff
Explorer

Hi, somesoni2
Yes you right, I need only EventData section.
I've try props.conf on heavy forwarder as you describe, but it doesnt work. Headers and footers don't remove.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

I've updated the regex for removefooter.

How are you getting the data into heavy forwarder, from universal forwarder? Were heavy forwarder restarted after making the change? Try keep these configurations on Universal forwarder.

0 Karma

borshoff
Explorer

I found what was wrong.
In input.conf we set "renderXml = true" . That's why props.conf doesn't apply to source::WinEventLog://ForwardedEvents. Cause source::WinEventLog://ForwardedEvents doesn't exist !

When i change it to "renderXml = false", filter start working!
But, i still need get this events in XML. Is there any way to do that?

"How are you getting the data into heavy forwarder, from universal forwarder?"
No, we collect all events by Windows Event collecor server in ForwardedEvents log. On the same VM we deploy heavy forwarder + Windows_TA addon with all necessary conf.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi borshoff,
you cannot filter your events to take only a part of them (only selected fields), You can filter events to take (or discard) all (full) events that match a regex, the only way to limit the dimensions of your events is to put a limit to the number of characters to take for each event (see limits.conf).
Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...