Getting Data In

Setting heavy forwarder for sending only few events which matched with Regx

moohkhol
New Member

I am setting up heavy forwarder on multiple machine, out of that one of them have below requirement,

1) Heavy forwarder should forward sub-set of events which match with given pattern and not all the events.

I search over web, I found couple of Q&A regarding the same but it’s seem not working for me.
First solutions I tried:

http://answers.splunk.com/answers/9076/how-to-configure-a-forwarder-to-filter-and-send-only-the-even...

On the above solution, I can see in forwarder log, it’s continuously saying could not connect with 0.0.0.0:0 and connection failing but nothing is getting forwarded at Indexer side.
Second solution:

I tried with forwarding everything into NullQueue and forward only event which match with REGX by setting up two stanza into transforms.conf.

Seems that above solution also does not work for me.

Pattern which I want to match is following,

oauth.googleusercontent.com, ssl.gstatic.com, fb.com, twitter.com

Heavy forwarder should only send events to indexer if events matched with any of the above patterns.

Please let me know, if you have any question regarding the same.
I have already spend one full day in exploring above but could not find solution.

Sorry to raise this question in that manner but i feel very sad after investigating over a day and could not find solution, Below are the configuration i am suing at my heavy forwarder side.

props.conf

[host::]
TRANSFORMS-set =setnull, allowtheseevents

transform.conf
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[allowtheseevents]
REGEX = (?m)ssl.gstatic.com
FORMAT = splunkindexer_9997

inputs.conf
[monitor:///usr/local/logfilename.log]
index = main
sourcetype = filtersplunkproxy
blacklist = .gz

output.conf
[tcpout]
defaultGroup = splunkindexer_9997

[tcpout:splunkindexer_9997]
server = :9997

[tcpout-server://:9997]
Please correct me if I am doing something wrong here.

Tags (1)
0 Karma

kristian_kolb
Ultra Champion

The essential problem lies in the [allowtheseevents] transform stanza. The FORMAT shall be indexQueue, not a reference to some outputs group.

http://docs.splunk.com/Documentation/Splunk/6.0.2/Forwarding/Routeandfilterdatad#Keep_specific_event...

Then you'll need to make your regex include more patterns.

/K

kristian_kolb
Ultra Champion

REGEX = (oauth\.googleusercontent|ssl\.gstatic|fb|twitter)\.com

Should work.

/k

0 Karma

moohkhol
New Member

Thanks Kristian, seems to be working this for me. If you don't mind, Please help me to have generate REGX for below pattern for OR condition with any one of oauth.googleusercontent.com,
ssl.gstatic.com,
fb.com,
twitter.com in anywhere in the events.

0 Karma

kristian_kolb
Ultra Champion

oh, and you need a DEST_KEY as well there;

[allowtheseevents]
REGEX = your regex
FORMAT = indexQueue
DEST_KEY = queue

/k

0 Karma

moohkhol
New Member

I have updated questions with config.

0 Karma

Ayn
Legend

Asking us to do all the work for you is in my eyes pretty rude. Provide us with info on your nullQueue setup with relevant sections from the conf files you were talking about yourself, then we can have a look and see where you might be going wrong.

Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...