Can I parse data at the indexer since I am using a Universal Forwarder on my source and how will that impact licensing?

[Magnus_001]

I am a using a Universal Forwarder on my domain controller to forward security events to a Splunk indexer and would like to filter out the static text "This event is generated..." in the security events to reduce the data consumption by the indexer. I know this can be done with a Heavy Forwarder at the source, but can I do this at the Indexer with the following entries in the props.conf and transforms.conf? Will it reduce my data consumption/license or is it too late by the time it reaches my indexer? Thanks!

Source: Windows Domain Controller (Universal Forwarder)

Splunk Indexer v6.1.x:

Props.conf

[WinEventLog:Security]
TRANSFORMS_ShortenMsg=ShortenSecMsg

Transforms.conf

[ShortenSecMsg]
REGEX=(?msi)(.*)This event is generated
DEST_KEY=_raw
FORMAT=$1

[Magnus_001]

Thank you all. I was able to filter out the unwanted static text with those changes to the Props.conf and Transforms.conf on the Indexer since I am using a UF on my DCs.

[martin_mueller]

I see. Based on the docs I'd have expected something different, will ask for docs clarification 🙂

[martin_mueller]

Did you try modifying inputs.conf on the UF to not include the text in the first place?

[Magnus_001]

Hi, the suppress_text = 1 works but it also removes many interesting fields we need (Account_Nmae, Account_domain, Logon_type, etc.).

[schose · Accepted solution]

Hi,

You can do it exactly as you mentioned. The props.conf and transforms.conf will be used on the indexer when "cooking" the data. So it's not to late to delete some data there before indexing. License meter is used after cooking the data. In this scenario a UF on the DC will be enough.

you will just need to define your the same sourcetype in inputs.conf on the UF as as you do in props.conf on the indexer. make sure all your props.conf are the same on all indexers.

Regards,

Andreas

[martin_mueller]

Replacing _raw isn't necessary, you can tell the UF to not even include that data.

[aweitzman]

To add to @schose's answer, your idea as posted is basically correct. However, since you are replacing _raw, you need to have a regex capture group for the content after the string you want to filter out, and include that in your format string, something like:

 [ShortenSecMsg]
 REGEX=^(.*)This event is generated(.*)$
 DEST_KEY=_raw
 FORMAT=$1$2

[martin_mueller]

Keep using a Universal Forwarder and check out the settings for WinEventLog type inputs at http://docs.splunk.com/Documentation/Splunk/6.2.2/admin/inputsconf - I'm no Windows expert, but you might just be looking for suppress_text = 1.

[fdi01]

use a heavy forwarder to do it .
because Universal Forwarder can not parse data .
http://docs.splunk.com/Splexicon:Heavyforwarder