I am being asked if we can remove some the extraneous data from our Windows logs at indexing time.
I remember in class the instructor talked about this, but I can't remember?
You can use SECMD in props.conf on the indexer/heavy forwarder to remove some of the extra text in the logs. Here are some examples.
https://answers.splunk.com/answers/44865/remove-out-section-of-log.html
https://answers.splunk.com/answers/4752/disabling-or-removing-extra-description-text-in-windows-2008...
https://gist.github.com/automine/5c8ef5b50e1df38249dfba01a70f2875
[WinEventLog:Security]
#Returns most of the space savings XML would provide
SEDCMD-clean0-null_sids = s/(?m)(^\s+[^:]+\:)\s+-?$/\1/g s/(?m)(^\s+[^:]+\:)\s+-?$/\1/g s/(?m)(\:)(\s+NULL SID)$/\1/g s/(?m)(ID\:)(\s+0x0)$/\1/g
SEDCMD-clean1-summary = s/This event is generated[\S\s\r\n]+$//g
SEDCMD-clean2-cert_summary = s/Certificate information is only[\S\s\r\n]+$//g
SEDCMD-clean3-blank_ipv6 = s/::ffff://g
SEDCMD-clean4-token_elevation_summary = s/Token Elevation Type indicates[\S\s\r\n]+$//g
SEDCMD-clean5-network_share_summary = s/(?ms)(A network share object was checked to see whether.*$)//g
SEDCMD-clean6-authentication_summary = s/(?ms)(The computer attempted to validate the credentials.*$)//g
SEDCMD-clean7-local_ipv6 = s/(?ms)(::1)//g
# Removed due to issue with Windows Filtering Platform events
# SEDCMD-clean8-firewall_summary = s/(?ms)(The Windows Filtering Platform has permitted.*$)//g
You sure can:
Whitelisting and blacklisting is what you are looking for!
hey There is one way to i.e. discard specific events and keep the rest
using props.conf and transforms.conf
As from the doc above follow Steps:
This example discards all sshd events in /var/log/messages by sending them to nullQueue:
In props.conf, set the TRANSFORMS-null attribute:
[source::/var/log/messages]
TRANSFORMS-null= setnull
Create a corresponding stanza in transforms.conf. Set DEST_KEY to "queue" and FORMAT to "nullQueue":
[setnull]
REGEX = [sshd]
DEST_KEY = queue
FORMAT = nullQueue
3.Restart Splunk Enterprise.
Let me know if this helps you!
Whilst this will work, this is not the most effective way to do this for windows logs - The windows event log input has a specific mechanism for dropping the large number of events windows produces, which prevents them being forwarded from the client - its far more efficient to drop them on the UF, than after they have crossed the network.