Filtering out WinEventLog events for a single user

[Lowell]

We have a monitoring system (WhatsUpGold) that periodically logs in to our windows machines and checks various conditions like low disk space or whether or not key services are up or down, and so on. This monitoring services creates tons of events in the Windows Security event log which splunk then indexes. The goal is to to completely filter out these events at index time. (This monitoring traffic accounts for the largest portion of our windows events logs for any given day.)

It's pretty easy to identify these logon events based on a SID and/or IP address of the monitoring system, however, that information is not contained in all of the events. The logout events have to be identified by the "Logon_ID" assigned in the login event. I would really like to filter out both the login, logout, and special privilege assignment events that occur each time the monitoring software polls one of our servers.

Has anyone else encountered a problem like this? Anyone find a suitable solution?

---

Update:

Here are the Windows Event types that I have to try to filter out:

EventCode=576   Special privileges assigned to new logon
EventCode=540   Successful Network Logon
EventCode=538   User Logoff
EventCode=4672  Special privileges assigned to new logon
EventCode=4624  An account was successfully logged on
EventCode=4634  An account was logged off

After looking close at the above events, I've determined that I was wrong when I said I could filter based on SID. In our current configuration I have to filter by source IP address, which is not in all of the above events.

But, as southeringtonp pointed out in a comment, the SID value is shown in all of the above events (which seems rather obvious to me now, but I had somehow overlooked this before). So I think I can get our domain admins to create monitoring-only account (and therefore unique SIDs) for all the automated activity, which then becomes a relatively simple matter of filtering out based on EventCode and SID.

I'll post an example as the answer when I get a working solution; just in case other people are looking to do something similar later.

[Lowell]

Our domain admin was able to quickly switch over most of the polling logins to use a new domain account. At that point this became a simple matter of filtering out a the set of eventcodes shown above and the specific SID used by the polling account.

Here is the configuration I ended up with.

Entry in props.conf:

[source::WinEventLog...]
# Really we only want to filter "Security" events; but we want the exact same stanza
# name as used in the system/default/props.conf to append to the settings given there
# rather than have a competing stanza name.
TRANSFORMS-drop_monitor_events = wineventlog_block_monitors

Entry in transforms.conf:

[wineventlog_block_monitors]
# Block various login/logout/special privs events caused by a polling system.
# This REGEX handles both Win2k3 and Win2k8 style events
#
# Blocking SID:    S-1-5-00-010144216-0301002201-931112210-2112
#
REGEX = (?ms)^[^ ]+ [^ ]+ [AP]M[\r\n]+LogName=Security[\r\n]+SourceName=[^\r\n]+[\r\n]EventCode=(?:538|540|576|4672|4624|4634)[\r\n].*?[\r\n](?:Sid=|\s+Security ID:\s+)S-1-5-00-010144216-0301002201-931112210-2112[\r\n]
DEST_KEY = queue
FORMAT = nullQueue

---

Additional notes:

(1) There are several differences introduced between Windows 2k3 events and Windows 2008. For starts new EventCodes are used (4xxx instead of the older 5xx codes). Also the older events would show the "SID" value on a line in header section as "Sid=<SID>", where as the newer-style events show this in the detail portion of the event on a line like "Security ID: <SID>" The regex above handles both of these scenarios.

Because the regex starts at the very beginning of the event, all non-security event and any events that don't match the specifically listed EventCode should cause the regex match to be discard quickly with minimal overhead. (There should be minimal regex overhead due to scanning and backtracking). Unfortunately, because of the fact that the 2008 style events can have multiple SIDs listed in a single event, we can't easily optimize the regex to give up quickly if the SID doesn't match; instead the regex engine has to search all the way to the end of the string looking for the matching SID line. I'm sure there's room for improvement here.

(2) We are only looking for specific EventCodes which are known to occur frequently, but any other kind of event we do want to come through. For example, if somebody logs in as our "monitor" domain user manually we want to be able to have some way to track that. For example, we had a case where one of our "monitor" user accounts logged in and installed a windows service (EventCode=4697)--now that is one event we really do want to have indexed in splunk.

[southeringtonp]

So, you want to keep them out of the index completely, not just keep them out of search results?

What event codes are you looking at? 538 or 4634 appear to include the name and sid of the account logging off as well as the Logon ID.