We just installed the universal forwarder on our domain controllers and are getting events just fine, however in most all of the Kerberos related events there is this junk text Windows puts in:
This event is generated when a logon session is created. It is generated on the computer that was accessed....
There are about 9 more sentences in there that are the same standard text across all the logs. Can this be removed either on the forwarder before indexing or on the indexer itself?
The SEDCMD-* option props.conf can be used to remove unwanted chunks of logentries.
http://docs.splunk.com/Documentation/Splunk/4.3.1/Data/Anonymizedatausingconfigurationfiles
For example:
[WinEventLog:Security]
SEDCMD-strip-detail-msg = s/(?ims)\s+^This event is generated .+//g
This has to be done at the event-processing instance. So if you're using an universal- or a lightweight forwarder, then you need to configure this on the indexer. In case of a heavy forwarder, you can strip the description at the source directly.
I was able to remove all of these verbose endings with just two SEDCMD statements:
SEDCMD-winsec1 = s/This event is generated.*[\r\n]+(.*[\r\n])+(.*$)?//
SEDCMD-winsec2 = s/Token Elevation Type indicates.*[\r\n]+(.*[\r\n])+(.*$)?//
The first one replaces your -4624, -4625, -4634, and -4769 statements, because in this format it does not matter how many lines follow the matched text.
good idea.
Here's what we put in place in our environment - thanks to the syntax I found here. 🙂 The numbers in the line refer to the event code.
SEDCMD-strip-detail-4688 = s/(?ims)Token Elevation Type indicates .*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*//g
SEDCMD-strip-detail-4624 = s/(?ims)This event is generated when.*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*//g
SEDCMD-strip-detail-4625 = s/(?ims)This event is generated when.*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*[\r\n].*//g
SEDCMD-strip-detail-4634 = s/(?ims)This event is generated when.*//g
SEDCMD-strip-detail-4769 = s/(?ims)This event is generated every time.*[\r\n].*[\r\n].*[\r\n].*[\r\n].*//g
If you want to find the extra lines in your data a search for "This event is generated" in the windows logs will point towards most of entries with extra lines.
The SEDCMD-* option props.conf can be used to remove unwanted chunks of logentries.
http://docs.splunk.com/Documentation/Splunk/4.3.1/Data/Anonymizedatausingconfigurationfiles
For example:
[WinEventLog:Security]
SEDCMD-strip-detail-msg = s/(?ims)\s+^This event is generated .+//g
This has to be done at the event-processing instance. So if you're using an universal- or a lightweight forwarder, then you need to configure this on the indexer. In case of a heavy forwarder, you can strip the description at the source directly.
It looks like your SEDCMD was correct. For some reason, it didn't like the source or sourcetype fields. When I replace it with the actual hostname, it worked just fine. Thanks!
Quote from docs:
Currently the following subset of
sed commands is supported:
* replace (s) and character substitution (y).
See http://docs.splunk.com/Documentation/Splunk/latest/admin/Propsconf
You should be able to simply use the example that is supplied in my answer. Not sure what your expression should actually do, but I think Splunk only supports replace and substituion mode (ie. s/some/thing/
y/some/thing/
)
Yes, this is what I'm looking for except I can't get it to work. I've tried your answer and also have tried:
[WinEventLog:Security]
SEDCMD-strip-detail-msg = /This event is generated when a logon session is created/,$d
sed '/This event is generated when a logon session is created/,$d' works just fine through a linux command, any recommendations?
The way to reduce your indexed volume and to get rid of useless events, is to setup filtering on the indexer.
Usually based on sourcetype, and a regex mach on the events.
See example in the docs http://docs.splunk.com/Documentation/Splunk/4.3.1/Deploy/Routeandfilterdatad#Filter_event_data_and_s...