Unable to filter WinEventLog inputs with RenderXml and XML character entities within pattern

[dstaulcu]

Filter attempts (whitelist or blacklist) on Message key value data appear to behave differently when renderXml = True compared to when renderXml = False.
Taking the following Event Message data for example:

 fragment_begin<Data Name='Image'>C:\Windows\System32\ping.exe</Data><Data Name='CommandLine'>fragment_end

When renderXml = False, the following expression succeeds in filtering events:

  blacklist = Message="<Data Name='Image'>.*\\(calc|ping).exe</Data><Data Name='CommandLine'>" 

However, when renderXml = True, the same expression fails to filter events.

After trying a various filtering strategies on this Message key/data when renderXml = True, it appears that matching fails any time when XML character entities (quote, ampersand, single quote, greater than, less than) are included in the pattern for matching.

I've tried cancelling these characters various ways (backslash, name, decimal) to no success.

Can anyone think of a workaround?

[cboggs]

I realize this is an old question - but I just had success filtering Windows Applocker event logs that have renderXml = true set, just by using the key's listed under the "Event Log whitelist and blacklist formats" listed [props.conf] docs1

In my case I wanted to filter out Applocker events where the file was allowed (to help us reduce license usage, because moving to XML increased the event size by 3 times on average). I tried several things, but finally just attempted the standard blacklist = EventCode="8002" and it worked!

So it would seem that the whitelist and blacklisting is still done on the plain text version of the event, not the XML one that actually gets forwarded down the pipeline.

Hope this helps!

[dstaulcu]

I spent some time thinking about how XML rendered event data would be represented in the Message key if not as a block of text containing the entirety of the XML schema.

At this point I think what is happening is that Message key represents a list object. List items in the list object likely contain xml entity value data. With that hypothesis, I figure Splunk enumerates list items in the list object for a match on a whitelist or blacklist pattern and forwards or suppresses events accordingly.

To validate an aspect of this theory, I updated my blacklist to filter out events containing an XML entity name (not data) that is common in all sysmon process creation events.

[WinEventLog://Microsoft-Windows-Sysmon/Operational]
blacklist = EventCode="1" Message="ParentImage"

This filter had no affect as I continued to receive events. While this does not support the entirety of my hypotheses, it does indicate that xml entity names are not included in the text evaluated by filters... and likely means that the XML characters are not included either.

[jkat54]

The XML version of windows events (rendered by windows not splunk) doesnt contain the "message" values.

See this post:
https://answers.splunk.com/answers/290844/testing-renderxml1-for-windows-event-logs-in-splun.html

You'll need to filter differently when using renderXML OR maybe you can make it work if you enable suppress_text=0 (defaults to this, so if you have suppress_text=1 change it to 0, and the look at the events for the "text".)

Below you see the xml view from windows app event log, and also a powershell view using get-eventlog (blue box). The powershell view shows a Message key value pair, the xml view does not. This is the same event, but different views of it.

[dstaulcu]

Thanks for the input.

Changing suppress_text either way did not positively affect the situation.

I too struggled with the idea of filtering on Message or EventCode keys when renderXml = True given that no such fields exist in Xml-rendered events. That said, basic filtering on those keys when renderXml = True does work as long as your filter expression does not contain xml formatting characters.

[jkat54]

My point is, when you render xml there isnt a field called Message...

  MachineName changes to <computer>, etc.

[jkat54]

So are you saying it works fine if you change it to something like

 blacklist = Message=".*ping.exe.*"

When renderXML=true

???

[jkat54]

Also just noticed I didnt put escape slash before .exe... nor did you. Have you tried escaping the dot?

 blacklist = Message="<Data Name='Image'>.*\\(calc|ping)\.exe</Data><Data Name='CommandLine'>" 

[dstaulcu]

Here are some filtering patterns I have tried which worked and did not work. In the list of things that worked you can see that escaping the dot in .exe was not needed.

#This works
#blacklist = EventCode="1" Message="notepad.exe"
#blacklist = EventCode="1" Message="CommandLine"   
#blacklist = EventCode="1" Message=":"
#blacklist = EventCode="1" Message="\\"
#blacklist = EventCode="1" Message="(notepad.exe|calc.exe)"   
#blacklist = EventCode="1" Message="(\\notepad.exe|calc.exe)"   
#blacklist = EventCode="1" Message="(?i)(\\notePad.exe|calc.exe)" 
#blacklist = EventCode="1" Message="(?i)\\(ping.exe|calc.exe)" 
#blacklist = EventCode="1" Message="(?i)\\(pIng|caLc).exe" 
#blacklist = EventCode="1" Message="(?i)\\(pIng|caLc).exe.*"

#Not working in Splunk
#blacklist = EventCode="1" Message="(?i)\\(pIng|caLc).exe\<"
#blacklist = EventCode="1" Message="(?i)\\(pIng|caLc).exe\<"
#blacklist = EventCode="1" Message="(?i)\\(pIng|caLc).exe\\<"
#blacklist = EventCode="1" Message="(?i)\\(pIng|caLc).exe\\\<"
#blacklist = EventCode="1" Message="(?i)\\(pIng|caLc).exe\\\\<"
#blacklist = EventCode="1" Message="'CommandLine'"