Getting Data In

Setting heavy forwarder for sending only few events which matched with Regx

moohkhol
New Member

I am setting up heavy forwarder on multiple machine, out of that one of them have below requirement,

1) Heavy forwarder should forward sub-set of events which match with given pattern and not all the events.

I search over web, I found couple of Q&A regarding the same but it’s seem not working for me.
First solutions I tried:

http://answers.splunk.com/answers/9076/how-to-configure-a-forwarder-to-filter-and-send-only-the-even...

On the above solution, I can see in forwarder log, it’s continuously saying could not connect with 0.0.0.0:0 and connection failing but nothing is getting forwarded at Indexer side.
Second solution:

I tried with forwarding everything into NullQueue and forward only event which match with REGX by setting up two stanza into transforms.conf.

Seems that above solution also does not work for me.

Pattern which I want to match is following,

oauth.googleusercontent.com, ssl.gstatic.com, fb.com, twitter.com

Heavy forwarder should only send events to indexer if events matched with any of the above patterns.

Please let me know, if you have any question regarding the same.
I have already spend one full day in exploring above but could not find solution.

Sorry to raise this question in that manner but i feel very sad after investigating over a day and could not find solution, Below are the configuration i am suing at my heavy forwarder side.

props.conf

[host::]
TRANSFORMS-set =setnull, allowtheseevents

transform.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[allowtheseevents]
REGEX = (?m)ssl.gstatic.com
FORMAT = splunkindexer_9997

inputs.conf
[monitor:///usr/local/logfilename.log]
index = main
sourcetype = filtersplunkproxy
blacklist = .gz

output.conf
[tcpout]
defaultGroup = splunkindexer_9997

[tcpout:splunkindexer_9997]
server = :9997

[tcpout-server://:9997]
Please correct me if I am doing something wrong here.

Tags (1)
0 Karma

kristian_kolb
Ultra Champion

The essential problem lies in the [allowtheseevents] transform stanza. The FORMAT shall be indexQueue, not a reference to some outputs group.

http://docs.splunk.com/Documentation/Splunk/6.0.2/Forwarding/Routeandfilterdatad#Keep_specific_event...

Then you'll need to make your regex include more patterns.

/K

kristian_kolb
Ultra Champion

REGEX = (oauth\.googleusercontent|ssl\.gstatic|fb|twitter)\.com

Should work.

/k

0 Karma

moohkhol
New Member

Thanks Kristian, seems to be working this for me. If you don't mind, Please help me to have generate REGX for below pattern for OR condition with any one of oauth.googleusercontent.com,
ssl.gstatic.com,
fb.com,
twitter.com in anywhere in the events.

0 Karma

kristian_kolb
Ultra Champion

oh, and you need a DEST_KEY as well there;

[allowtheseevents]
REGEX = your regex
FORMAT = indexQueue
DEST_KEY = queue

/k

0 Karma

moohkhol
New Member

I have updated questions with config.

0 Karma

Ayn
Legend

Asking us to do all the work for you is in my eyes pretty rude. Provide us with info on your nullQueue setup with relevant sections from the conf files you were talking about yourself, then we can have a look and see where you might be going wrong.

Get Updates on the Splunk Community!

Introducing Splunk Enterprise 9.2

WATCH HERE! Watch this Tech Talk to learn about the latest features and enhancements shipped in the new Splunk ...

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...