Can you Filter on remote servers prior to the log transfers to Splunk Indexer?

[agentguerry]

Can I set up props.conf and transform.conf files on a remote server to null out entries in a log prior to them getting sent to the splunk index servers?

I want to filter out any lines in my /var/log/messages that contains:
com.site.ws.util.jaxws.SOAPMessageContextUtil - Could not instantiate EfficientStreamMessage java.lang.NoSuchFieldException: envelopeTag

I created a props and transforms file:

[source::/var/log/messages]
TRANSFORMS-null= setnull

[setnull]
REGEX = com.site.ws.util.jaxws.SOAPMessageContextUtil - Could not instantiate EfficientStreamMessage java.lang.NoSuchFieldException: envelopeTag
DEST_KEY = queue
FORMAT = nullQueue

Restarted Splunk.
But the Splunk Web interface still shows them being logged.

Thanks for the help in advance. I feel I'm overlooking something simple. 😄

[masonmorales]

As others mentioned, you need to replace your current universal forwarder with a heavy forwarder. You can copy all of your configuration files from the UF to the HF without having to rewrite anything. Then, have a look at this article on how to configure it: http://docs.splunk.com/Documentation/Splunk/6.2.2/Forwarding/Routeandfilterdatad

[s2_splunk]

OR: leave your universal forwarder installed and deploy your props/transforms onto your indexer. Granted, you will still send the data over the network, but you don't have the footprint and resource requirements of a full Splunk install (Heavy Forwarder) taking away capacity on your remote server.

BTW, you can't do this kind of event level filtering, because the UF doesn't know anything about events, it sends a data stream that's parsed and line-broken on the indexer (or a heavy forwarder), which is when indexed events get created.

[masonmorales]

In addition to wasting network bandwidth, it also consumes CPU resources on your indexer. If an HF can be put at the sending device, that's what should be done.

[agentguerry]

So i am reading that you ahve to have the "heavy forwarder" in order to filter in that manner.

[martin_mueller]

Yup, Heavy Forwarders can process their data before sending it to the indexers.