Hello, all.
I know that my question's not a unique, but I want to ask it 🙂
I have a netflow text log on a server with a universal forwarder installed.
I don't want to index this entire log. I only want to index fields containing a certain key. For example, I can provide a few strings:
{"timestamp":"2016-11-22T15:42:17.037821+0300","flow_id":268878859621513,"event_type":"netflow","src_ip":"11.11.11.11","src_port":22,"dest_ip":"22.22.22.22","dest_port":44206,"proto":"TCP","app_proto":"ssh","netflow":{"pkts":8,"bytes":2230,"start":"2016-11-22T15:41:14.611465+0300","end":"2016-11-22T15:41:14.638311+0300","age":0},"tcp":{"tcp_flags":"1a","syn":true,"psh":true,"ack":true}}
{"timestamp":"2016-11-22T15:44:18.013133+0300","flow_id":720902685008782,"event_type":"netflow","src_ip":"157.55.130.156","src_port":40032,"dest_ip":"22.22.22.22","dest_port":3166,"proto":"UDP","netflow":{"pkts":2,"bytes":126,"start":"2016-11-22T15:39:17.402318+0300","end":"2016-11-22T15:39:17.527073+0300","age":0}}
{"timestamp":"2016-11-22T15:44:16.025489+0300","flow_id":265292561318767,"event_type":"netflow","src_ip":"22.22.22.22","src_port":41979,"dest_ip":"33.33.33.33","dest_port":443,"proto":"TCP","app_proto":"tls","netflow":{"pkts":40,"bytes":14432,"start":"2016-11-22T15:41:05.983919+0300","end":"2016-11-22T15:43:14.286741+0300","age":129},"tcp":{"tcp_flags":"1b","syn":true,"fin":true,"psh":true,"ack":true}}
As you can see, we have a different field - proto and app_proto. I only want to index data with these specific fields in Splunk. For example, I only need events with proto":"TCP", or maybe proto":"TCP" and (or) app_proto":"ssh"
Can you help my with this case? I read the manual, but I can't understand the principle of the implementation of this.
Thanks!
Hi templier,
for my knowledge, you can filter your events discarding those events that don't contain your strings, but it isn't possible take only a part of each event that contains one of your strings.
Every way to take only events that contain your strings, you have to configure:
props.conf
[your_sourcetype]
TRANSFORMS-set-nullqueue=set_nullqueue,set_OK
transforms.conf
[set_nullqueue]
REGEX=.
DEST_KEY=queue
FORMAT=nullQueue
[set_OK]
REGEX=regex1|regex2|regex3
DEST_KEY = queue
FORMAT = indexQueue
Bye.
Giuseppe
Hi templier,
Filtering using props.conf and transforms.conf it's a part of parsing phase done on your indexers.
It isn't a good idea to use an heavy forwarder in all your servers!
Bye.
Giuseppe
Hi templier,
for my knowledge, you can filter your events discarding those events that don't contain your strings, but it isn't possible take only a part of each event that contains one of your strings.
Every way to take only events that contain your strings, you have to configure:
props.conf
[your_sourcetype]
TRANSFORMS-set-nullqueue=set_nullqueue,set_OK
transforms.conf
[set_nullqueue]
REGEX=.
DEST_KEY=queue
FORMAT=nullQueue
[set_OK]
REGEX=regex1|regex2|regex3
DEST_KEY = queue
FORMAT = indexQueue
Bye.
Giuseppe
What do you mean when you say props.conf unchanged: Do you used my props.conf?
Male this test inverting Order in TRANSFORMS command TRANSFORMS-set-nullqueue=set_OK,set_nullqueue
Are you sure of your regex?
Bye.
Giuseppe
Exactly, this needs to be done on a heavy forwarder.
If interested, I would adjust the regular expression:
transforms.conf
[setnull]
REGEX = (\"proto\":\"UDP\")
DEST_KEY = queue
FORMAT = nullQueue
[setok]
REGEX = (\"proto\":\"TCP\")|(\"app_proto\":\"ssh\")
DEST_KEY = queue
FORMAT = nullQueue
props.conf
[your_sourcetype]
TRANSFORMS-set = setnull, setok
Rodrigo Ribeiro
Hello,
It's work.
And now I have more experience in this theme.
Can you to issue this post as an Answer, rather than a comment? I mark it 🙂
Many thanks.
If you like, accept my answer.
Thank you.
Bye.
Giuseppe
Hi templier,
Filtering using props.conf and transforms.conf it's a part of parsing phase done on your indexers.
It isn't a good idea to use an heavy forwarder in all your servers!
Bye.
Giuseppe
Perfect Cusello.
As informed by our colleague cusello, can be done by indexers, but in fact can not be done in a universal forwarder 🙂
Tks
Rodrigo Ribeiro
Yeap, thanks to you and Giuseppe for information and live example of this solution
I test it on splunk field extraction - work great.
Maybe solution in uninstall universal forwarder and install a heavy forwarder?
Hello, Giuseppe.
Thx for you answer.
Tried do this, my files:
transforms.conf
[set_nullqueue]
REGEX=\S*UDP\S*
DEST_KEY=UDP
FORMAT=nullQueue
[set_OK]
REGEX=\S*ssh\S*
DEST_KEY = queue
FORMAT = indexQueue
props.conf unchanged, only set my sourcetype
And nothing new in result. I write to indexer lines contained UDP
A space in regex is \s not \S, try replacing that.
Try, nothing new