I have syslog data coming to a distributed environment. I am trying to send the data to a specific index based on a string that is in the data. The string is "NetScreen" and the index is juniper. I have the following props.conf and transforms.conf deployed to the forwarder that is collecting these logs but the logs are still coming into index main
Props.conf
[JuniperFW]
Transforms-JuniperFW=JuniperFW
Transforms.conf
[JuniperFW]
DEST_KEY = _MetaData:Index
REGEX = (NetScreen)
FORMAT = index::juniper
"The forwarder that is collecting these logs" -> Is this a Universal Forwarder or a Heavy Forwarder? The UF only does input phase. In which case you would have to set this on your Indexer(s) or Heavy Forwarders. See the Splunk Wiki for a handy reference for what applies where.
Also you probably want to capitalize TRANSFORMS in the props.conf
"The forwarder that is collecting these logs" -> Is this a Universal Forwarder or a Heavy Forwarder? The UF only does input phase. In which case you would have to set this on your Indexer(s) or Heavy Forwarders. See the Splunk Wiki for a handy reference for what applies where.
Also you probably want to capitalize TRANSFORMS in the props.conf
Do you know if I can also change the sourcetype here. I have tried using another stanza in transforms.conf and props.conf as shown below, but it is not changing the sourcetype, is there another method I should be using?
props.conf
[JuniperFW]
TRANSFORMS-JuniperFW=JuniperFW
[JuniperFW_ST]
TRANSFORMS-JuniperRW_ST=JuniperFW_ST
transforms.conf
[JuniperFW]
DEST_KEY = _MetaData:Index
REGEX = (NetScreen)
FORMAT = juniper
[JuniperFW_ST]
DEST_KEY = MetaData:Sourcetype
REGEX = (NetScreen)
FORMAT = JuniperFW
Also related: http://splunkreactions.tumblr.com/post/87038427475 😄
You should be able to change sourcetype at index time as well... (I'm not sure if the other sourcetype's props would apply or not when you change it, I haven't done much in this space). But I will note that according to the transforms.conf doc setting MetaData:Sourcetype requires the value to be prefixed sourcetype::
so your FORMAT
line should be sourcetype::JuniperFW
Alternatively, you can also rename sourcetypes at search time (slightly different configs).
I ended up leaving them in index=main and then setting the sourcetype and index time. My ultimate goal was to normalize the data to follow the CIM so I could create searches across multiple types of firewalls. With the data separated into sourcetypes I am now able to create the search time field extractions I needed, although I struggled with this until I realized that the app that has the field extractions needed to be set to be shared globally in metadata/default.meta.
Everything is working now and now its time to start actually using the data!!
It is a UF...
Moving those to the Indexer took care of it! I had to change FORMAT = index:juniper to FORMAT = juniper but that got the data going to right location. Thank you for your help!