Hello - I want to send only events with keyword BIDPRICE from my application logs. I guess i need to modifiy props.conf, transforms.conf and outputs.conf
Can someone help me what changes are acually needed.
fyi-Currently forwading is working fine without this filter.
Thanks in advance for help.
This is quite thoroughly discussed in the docs, please see:
http://docs.splunk.com/Documentation/Splunk/4.3.1/Deploy/Routeandfilterdatad#Keep_specific_events_an...
If you have a heavy forwarder, the settings should be placed there. In all other cases, the settings should be on the indexer.
In short, what you need to do is
In props.conf:
[your_applog_sourcetype]
TRANSFORMS-keep_only_bidprice= setnull,setparsing
In transforms.conf:
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = BIDPRICE
DEST_KEY = queue
FORMAT = indexQueue
UPDATE:
What this does is to apply a set of rules to your incoming data. First, in props.conf you tell splunk that all events of a certain sourcetype should pass through a transform (or in this case, two transforms - setnull
and setparsing
).
Then, in transforms.conf, you state how data should be treated. setnull
routes data to the nullQueue, i.e. throws it away. setparsing
will match all events containing the string BIDPRICE and send these on for parsing and indexing.
If this does not work for you, then please tell us more about your setup, and post the relevant parts of props.conf and transforms.conf, along with a few sample events from your application log file.
You should know though that this will only work for new data coming in, and not alter any existing events already in your index.
UPDATE AGAIN:
DEST_KEY = queue
is where you state what parameter should be altered by the transform.
FORMAT = nullQueue
is the value that will be set for the parameter. (nullQueue
is something like /dev/null
)
Think of it as saying queue = nullQueue
.
So what happens when use TRANSFORMS-set = setnull, setparsing
in props.conf is;
set the destination of all events to
nullQueue (since all events will
match the dot (.) in the regex, i.e.
throw them away.
then, for those matching the regex in
setparsing
, i.e.
BidPrice, the destination should be
re-written to be the indexQueue,
which is where events normally for
indexing.
Hope this helps,
Kristian
This is quite thoroughly discussed in the docs, please see:
http://docs.splunk.com/Documentation/Splunk/4.3.1/Deploy/Routeandfilterdatad#Keep_specific_events_an...
If you have a heavy forwarder, the settings should be placed there. In all other cases, the settings should be on the indexer.
In short, what you need to do is
In props.conf:
[your_applog_sourcetype]
TRANSFORMS-keep_only_bidprice= setnull,setparsing
In transforms.conf:
[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue
[setparsing]
REGEX = BIDPRICE
DEST_KEY = queue
FORMAT = indexQueue
UPDATE:
What this does is to apply a set of rules to your incoming data. First, in props.conf you tell splunk that all events of a certain sourcetype should pass through a transform (or in this case, two transforms - setnull
and setparsing
).
Then, in transforms.conf, you state how data should be treated. setnull
routes data to the nullQueue, i.e. throws it away. setparsing
will match all events containing the string BIDPRICE and send these on for parsing and indexing.
If this does not work for you, then please tell us more about your setup, and post the relevant parts of props.conf and transforms.conf, along with a few sample events from your application log file.
You should know though that this will only work for new data coming in, and not alter any existing events already in your index.
UPDATE AGAIN:
DEST_KEY = queue
is where you state what parameter should be altered by the transform.
FORMAT = nullQueue
is the value that will be set for the parameter. (nullQueue
is something like /dev/null
)
Think of it as saying queue = nullQueue
.
So what happens when use TRANSFORMS-set = setnull, setparsing
in props.conf is;
set the destination of all events to
nullQueue (since all events will
match the dot (.) in the regex, i.e.
throw them away.
then, for those matching the regex in
setparsing
, i.e.
BidPrice, the destination should be
re-written to be the indexQueue,
which is where events normally for
indexing.
Hope this helps,
Kristian
you are welcome 🙂
It works now ... (relief). I guess the problem was within props.conf file. Looks like [host:: XXXX] does not work with universal forwaders. I chaged this to [source::
Thanks a lot for your guidence
Universal Forwarder -> indexer.
Sorry, but WHERE (in which file, on which host) are you making these configuration changes, and what is your setup?
A) Heavy Forwarder -> indexer ?
B) Universal Forwarder -> indexer ?
For the rest, see update above.
/k
Does not work at all if I don't add _TCP_ROUTING. I have added this because document says for TCP routing.(between two Linux servers).
What does desk_key = queue mean?
My setup is still not okay, I think reason is [setnull] options around.
Thanks for your help.
And don't use _TCP_ROUTING
unless you know what you are doing.
Then the regex in transforms.conf should look like;
REGEX = BidPrice
/k
What does the event look like? You have to make sure that the regex matches the text in the event. If it does not, then all events are thrown away - since they match the setnull transform.
/k
Sorry for conusion around. Here is my current setup wrt Filtering and i see all events flowing down to Indexer
*props.conf *
[host::sgppsr00346.XXXX.XXXX.com]
TRANSFORMS-set= setnull,setparsing
*transforms.conf *
[setnull]
REGEX = .
DEST_KEY = queue [Not sure what should be here for the case of TCP]
FORMAT = nullQueue [Not sure what should be here for the case of TCP]
[setparsing]
REGEX = [BidPrice]
DEST_KEY = _TCP_ROUTING [TCP routing]
FORMAT = GroupName [This is currect group name as per outputs.conf]
see update above. /k
Thanks for your reply But This did not work.. I still see all the events in my indexer (Indexer was restarted). Not sure what below values refers to, Any conf update needed to understand what is queue and nullQueue?
"
DEST_KEY = queue
FORMAT = nullQueue
"
If these are light or universal forwarder, you cannot filter the logs there. You'll have to do it at the indexer instead. Instructions on how to do this are available in the docs: http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Routeandfilterdatad#Filter_event_data_and_...