Getting Data In

Filtering events from forwarder at indexer

twinspop
Influencer

I'm trying to filter noisy events that have recently pushed us over license usage. The events come from a lightweight forwarder that I don't have access to at the moment. I'd like to filter at the indexer. This is not working:

props.conf:

[source::/Logs/Noisy.log]
TRANSFORMS-set = setnull

transforms.conf:

[setnull]
REGEX = host07
DEST_KEY = queue
FORMAT = nullQueue

Where "host07" is present in all the logs I want to dump, and the source matches that displayed in search results on the indexer. I've also tried listing the sourcetype as shown in search results.

Gotta be something simple... or maybe LWF traffic needs to be filtered on the LWF, not the indexer?

Thanks, Jon

Tags (1)
0 Karma

the_wolverine
Champion

Are you sure its a LWF? If not, the data is already cooked and cannot be manipulated at the indexer.

0 Karma

twinspop
Influencer

Is there any way to tell from the indexer's side? I did not do the install, just gave a list of command line instructions. It's possible the server admin didn't follow them.

0 Karma

dwaddle
SplunkTrust
SplunkTrust

Lightweight Forwarder requires filtering to be done at the indexer -- it cannot do it itself.

The answer http://splunk-base.splunk.com/answers/96/how-do-i-exclude-some-events-from-being-indexed-by-splunk has some good examples of using nullQueue.

At a quick glance, your example looks fundamentally correct. Does "host07" appear in the _raw for the events in question, or is it in another metadata field?

0 Karma

twinspop
Influencer

Yes, the string is in _raw.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...