Filter events from forwards

[chittari]

Hello - I want to send only events with keyword BIDPRICE from my application logs. I guess i need to modifiy props.conf, transforms.conf and outputs.conf

Can someone help me what changes are acually needed.
fyi-Currently forwading is working fine without this filter.

Thanks in advance for help.

[kristian_kolb · Accepted solution]

This is quite thoroughly discussed in the docs, please see:
http://docs.splunk.com/Documentation/Splunk/4.3.1/Deploy/Routeandfilterdatad#Keep_specific_events_an...

If you have a heavy forwarder, the settings should be placed there. In all other cases, the settings should be on the indexer.

In short, what you need to do is

1. In props.conf:

   [your_applog_sourcetype]
TRANSFORMS-keep_only_bidprice= setnull,setparsing

2. In transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = BIDPRICE
DEST_KEY = queue
FORMAT = indexQueue

---

UPDATE:

What this does is to apply a set of rules to your incoming data. First, in props.conf you tell splunk that all events of a certain sourcetype should pass through a transform (or in this case, two transforms - setnull and setparsing).

Then, in transforms.conf, you state how data should be treated. setnull routes data to the nullQueue, i.e. throws it away. setparsing will match all events containing the string BIDPRICE and send these on for parsing and indexing.

If this does not work for you, then please tell us more about your setup, and post the relevant parts of props.conf and transforms.conf, along with a few sample events from your application log file.

You should know though that this will only work for new data coming in, and not alter any existing events already in your index.

---

UPDATE AGAIN:

DEST_KEY = queue is where you state what parameter should be altered by the transform.

FORMAT = nullQueue is the value that will be set for the parameter. (nullQueue is something like /dev/null)

Think of it as saying queue = nullQueue.

So what happens when use TRANSFORMS-set = setnull, setparsing in props.conf is;

• set the destination of all events to
  nullQueue (since all events will
  match the dot (.) in the regex, i.e.
  throw them away.

• then, for those matching the regex in
  setparsing, i.e.
  BidPrice, the destination should be
  re-written to be the indexQueue,
  which is where events normally for
  indexing.

Hope this helps,

Kristian

[kristian_kolb]

you are welcome 🙂

[chittari]

It works now ... (relief). I guess the problem was within props.conf file. Looks like [host:: XXXX] does not work with universal forwaders. I chaged this to [source:: ] and it worked.

Thanks a lot for your guidence

[chittari]

Universal Forwarder -> indexer.

[kristian_kolb]

Sorry, but WHERE (in which file, on which host) are you making these configuration changes, and what is your setup?

A) Heavy Forwarder -> indexer ?
B) Universal Forwarder -> indexer ?

For the rest, see update above.

/k

[chittari]

Does not work at all if I don't add _TCP_ROUTING. I have added this because document says for TCP routing.(between two Linux servers).

What does desk_key = queue mean?

My setup is still not okay, I think reason is [setnull] options around.

Thanks for your help.

[kristian_kolb]

And don't use _TCP_ROUTING unless you know what you are doing.

[kristian_kolb]

Then the regex in transforms.conf should look like;

REGEX = BidPrice

/k

[chittari]

Event looks like this

INFO 19:55:55,284 - UST prices for FIDO : [423423] : BidPrice[103.25390625] AskPrice[103.28515625] BidYield[0.7235491957] AskYield[0.71622942792] at [1335354955284]

[kristian_kolb]

What does the event look like? You have to make sure that the regex matches the text in the event. If it does not, then all events are thrown away - since they match the setnull transform.

/k

[chittari]

Sorry for conusion around. Here is my current setup wrt Filtering and i see all events flowing down to Indexer

*props.conf *

[host::sgppsr00346.XXXX.XXXX.com]
TRANSFORMS-set= setnull,setparsing

*transforms.conf *

[setnull]
REGEX = .
DEST_KEY = queue [Not sure what should be here for the case of TCP]
FORMAT = nullQueue [Not sure what should be here for the case of TCP]

[setparsing]
REGEX = [BidPrice]
DEST_KEY = _TCP_ROUTING [TCP routing]
FORMAT = GroupName [This is currect group name as per outputs.conf]

[kristian_kolb]

see update above. /k

[chittari]

Thanks for your reply But This did not work.. I still see all the events in my indexer (Indexer was restarted). Not sure what below values refers to, Any conf update needed to understand what is queue and nullQueue?
"
DEST_KEY = queue
FORMAT = nullQueue
"

[Ayn]

If these are light or universal forwarder, you cannot filter the logs there. You'll have to do it at the indexer instead. Instructions on how to do this are available in the docs: http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Routeandfilterdatad#Filter_event_data_and_...